Image motion estimation method and related device

Abstract

The invention provides an image motion estimation method and a related device, and an electronic device can intercept a rendering instruction for an image frame, obtains the related resources of the image frame based on the rendering instruction, carries out the motion estimation of two adjacent image frames through the fusion of an optical flow motion estimation method and a 3D motion field motion estimation method, and achieves the motion estimation of the two adjacent image frames. And determining motion vector information between two adjacent image frames. In this way, the motion estimation effect between two adjacent image frames can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image motion estimation method and related devices. Background Art

[0002] The rapid development of mobile terminals and mobile Internet has brought great convenience to people's lives. Games, videos and other applications have become important ways for people to work, relax and entertain. In order to improve the picture smoothness of games, videos and other applications, it is necessary to perform motion estimation on the original image frame to obtain motion information. Then, based on the motion information and the original image frame, a new target image frame is generated. Therefore, the motion estimation effect of the original image frame is closely related to the generation quality of the target image frame. At present, the motion estimation effect of the motion estimation algorithm is poor, resulting in low generation quality of the target image frame. Summary of the invention

[0003] The present application provides an image motion estimation method and related device, which can realize the fusion of optical flow motion estimation method and 3D motion field motion estimation method, perform motion estimation on two adjacent image frames, and determine the motion vector information between the two adjacent image frames. In this way, the accuracy of the motion estimation result between the two adjacent image frames can be improved.

[0004] In a first aspect, the present application provides an image motion estimation method, which is applied to an electronic device, wherein the electronic device has a first application installed, and the method comprises: obtaining a rendering instruction issued by the first application, wherein the rendering instruction is used to instruct a graphics processor GPU to render an image frame of the first application, wherein the image frame of the first application comprises a first image frame and a second image frame; identifying from the rendering instruction a pixel position of a first object in the first image frame and a pixel position of a second object in the first image frame; wherein the first object and the second object are of different types; identifying from the rendering instruction a pixel position of the first object in the second image frame and a pixel position of the second object in the second image frame; determining, by an optical flow motion estimation method, a motion vector of a pixel position of the first object between the first image frame and the second image frame; obtaining native motion vector information of the first application, and determining from the native motion vector information a motion vector of a pixel position of the second object between the first image frame and the second image frame.

[0005] Through an image motion estimation method provided by the present application, different motion estimation methods can be used for different types of rendering objects by intercepting the rendering instructions of the application for each rendering object in the image frame. In this way, different motion estimation methods are applicable to different rendering objects. For example, the effect of the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. However, when the movement speed of the object in the picture frame is fast, the result of the optical flow motion estimation will have errors. The 3D motion field motion estimation method can also ensure more accurate motion estimation results when the movement speed of the object in the image frame is fast. However, the 3D motion field motion estimation method cannot perform motion estimation on translucent objects. The native motion field vector information of the application is determined by the 3D motion field motion estimation method. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can improve the accuracy of the motion estimation results between two adjacent image frames.

[0006] In one possible implementation, the electronic device includes a motion estimation module; the obtaining of the rendering instruction issued by the first application specifically includes: the motion estimation module obtaining the rendering instruction issued by the first application from a graphics interface; the obtaining of native motion vector information of the first application, and determining the motion vector of the pixel position of the second object between the first image frame and the second image frame from the native motion vector information, specifically includes: the motion estimation module obtaining the native motion vector information of the first application from the graphics interface, and determining the motion vector of the pixel position of the second object between the first image frame and the second image frame from the native motion vector information.

[0007] In this way, the native motion vector information of the application can be directly obtained from the graphics interface, so that the application does not need to be adapted and the application is unaware of it, and the native motion vector information can be obtained.

[0008] In a possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector information between the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; the motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

[0009] In this way, the optical flow algorithm interface on the system on chip (SoC) platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0010] In a possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame based on the pixel position of the first object in the first image frame, the pixel position of the second object in the second image frame, the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame to the motion estimation module; the motion estimation module uses the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame as the motion vector of the pixel position of the first object between the first image frame and the second image frame.

[0011] In this way, the optical flow algorithm interface on the system-on-chip platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0012] In a possible implementation, the method also includes: the motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; the motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; the optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

[0013] In a possible implementation manner, the native motion vector information is motion field motion vector information between the first image frame and the second image frame determined by the first application through a 3D motion field estimation method.

[0014] In a possible implementation, the electronic device also includes an interpolation module; the method also includes: the motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; the interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

[0015] In this way, the motion vector information between the first image frame and the second image frame can be used to perform frame interpolation, thereby saving power consumption of the electronic device.

[0016] In one possible implementation, before identifying the pixel position of the first object in the first image frame and the pixel position of the second object in the first image frame from the rendering instruction, the method also includes: the motion estimation module determines that the native motion vector information can be obtained from the graphics interface.

[0017] In a possible implementation, the optical flow motion estimation method includes any one of the following: a pyramid optical flow method, a Lucas-Kanad method, and an optical flow method based on a neural network.

[0018] In a possible implementation manner, the graphics interface may include any one of the following: an open graphics library OpenGL, an open graphics library OpenGL ES for embedded systems, and a drawing application program interface Vulkan.

[0019] In a second aspect, the present application provides an image motion estimation method, which is applied to an electronic device, wherein the electronic device is installed with a first application; the method comprises: obtaining a rendering instruction issued by the first application, wherein the rendering instruction is used to instruct a GPU to render an image frame of the first application, wherein the image frame of the first application comprises a first image frame and a second image frame; identifying from the rendering instruction the pixel position of a first object in the first image frame and the pixel position of a second object in the first image frame; identifying from the rendering instruction the pixel position of the first object in the second image frame and the pixel position of the second object in the second image frame; determining, by means of an optical flow motion estimation method, a motion vector of a pixel position of the first object between the first image frame and the second image frame; obtaining a first transformation matrix and first depth information of the first image frame, and a second transformation matrix and second depth information of the second image frame; and determining, by means of a 3D motion field motion estimation method, a motion vector of a pixel position of the second object between the first image frame and the second image frame.

[0020] Through an image motion estimation method provided by the present application, different motion estimation methods can be used for different types of rendering objects by intercepting the rendering instructions of the game application for each rendering object in the image frame. In this way, different motion estimation methods are applicable to different rendering objects. For example, the effect of the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. The 3D motion field motion estimation method can also ensure more accurate motion estimation results when the object in the image frame moves faster. However, the motion vector of the dynamic object is relatively cumbersome to calculate using the 3D motion field estimation method, and the calculation time is relatively long. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can quickly determine the motion estimation results between two adjacent image frames.

[0021] In one possible implementation, the electronic device includes a motion estimation module; the obtaining of the rendering instruction issued by the first application specifically includes: the motion estimation module obtaining the rendering instruction issued by the first application from a graphics interface; the obtaining of the first transformation matrix and first depth information of the first image frame, and the second transformation matrix and second depth information of the second image frame specifically includes: the motion estimation module obtaining the first transformation matrix and first depth information of the first image frame, and the second transformation matrix and second depth information of the second image frame from the graphics interface.

[0022] In this way, the transformation matrix and depth information of the image frame in the application can be directly obtained from the graphics interface, so that the application does not need to adapt and the application is unaware of the need to obtain the transformation matrix and depth information of the image frame.

[0023] In a possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector information between the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; the motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

[0024] In this way, the optical flow algorithm interface on the system-on-chip platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0025] In a possible implementation, the electronic device also includes an optical flow algorithm interface; the optical flow motion estimation method is used to determine the motion vector of the pixel position of the first object between the first image frame and the second image frame, specifically including: the optical flow algorithm interface determines the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame based on the pixel position of the first object in the first image frame, the pixel position of the first object in the second image frame, the first image frame and the second image frame; the optical flow algorithm interface sends the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame to the motion estimation module; the motion estimation module uses the optical flow motion vector of the pixel position of the first object between the first image frame and the second image frame as the motion vector of the pixel position of the first object between the first image frame and the second image frame.

[0026] In this way, the optical flow algorithm interface on the system-on-chip platform can be directly called to perform motion estimation through the optical flow motion estimation method, which simplifies the computational complexity of the motion estimation module.

[0027] In a possible implementation, the method also includes: the motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; the motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; the optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

[0028] In a possible implementation, the electronic device also includes an interpolation module; the method also includes: the motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; the interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

[0029] In this way, the motion vector information between the first image frame and the second image frame can be used to perform frame interpolation, thereby saving power consumption of the electronic device.

[0030] In a possible implementation, the optical flow motion estimation method includes any one of the following: a pyramid optical flow method, a Lucas-Kanad method, and an optical flow method based on a neural network.

[0031] In a possible implementation manner, the graphics interface may include any one of the following: an open graphics library OpenGL, an open graphics library OpenGL ES for embedded systems, and a drawing application program interface Vulkan.

[0032] In a third aspect, the present application provides an image motion estimation method, which is applied to an electronic device, and the electronic device is installed with a first application; the method includes: obtaining a rendering instruction issued by the first application, the rendering instruction is used to instruct the GPU to render an image frame of the first application, and the image frame of the first application includes a first image frame and a second image frame; obtaining a first transformation matrix of the first image frame, a first depth information of the first image frame, a second transformation matrix of the second image frame, and a second depth information of the second image frame from the graphics interface; determining the optical flow motion vector information between the first image frame and the second image frame through an optical flow motion estimation method; the optical flow motion vector information includes an optical flow motion vector at a first pixel position in the second image; based on the first transformation matrix, the first depth information, the second transformation matrix, and the second depth information, determining the motion field motion vector information between the first image frame and the second image frame through a 3D motion field motion estimation method; the optical flow motion vector information includes a motion field motion vector at the first pixel position; if the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is less than a first specified threshold, taking the average of the optical flow motion vector at the first pixel position and the motion field vector at the first pixel position as the motion vector at the first pixel position.

[0033] The image motion estimation method provided by the present application can realize motion estimation of two adjacent image frames by integrating the transformation matrix and depth information of two adjacent image frames with the optical flow motion estimation method and the 3D motion field motion estimation method, and determine the motion vector information between the two adjacent image frames. In this way, the accuracy of the motion estimation result can be improved.

[0034] In a possible implementation, if the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is greater than or equal to a first specified threshold, the depth value of the first pixel position and the depth values ​​of the surrounding pixel positions around the first pixel position in the second image frame are obtained from the second depth information; if the depth value at the first pixel position differs from the depth value of the surrounding pixel positions by less than a second specified threshold, the optical flow motion vector at the first pixel position is used as the motion vector at the first pixel position; if the depth value at the first pixel position differs from the depth value of the surrounding pixel positions by less than a second specified threshold, the motion field motion vector at the first pixel position is used as the motion vector at the first pixel position.

[0035] In a possible implementation, the optical flow motion estimation method includes any one of the following: a pyramid optical flow method, a Lucas-Kanad method, and an optical flow method based on a neural network.

[0036] In a possible implementation manner, the graphics interface may include any one of the following: an open graphics library OpenGL, an open graphics library OpenGL ES for embedded systems, and a drawing application program interface Vulkan.

[0037] In a fourth aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program, so that when the one or more processors execute the computer program, the method in any possible implementation of the above-mentioned first aspect is executed.

[0038] In a fifth aspect, the present application provides another electronic device, comprising one or more functional modules, wherein the one or more functional modules are used to execute the method in any possible implementation of the first aspect above.

[0039] In a sixth aspect, the present application provides a chip system, comprising a processing circuit and an interface circuit, wherein the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method in any possible implementation of any of the above aspects.

[0040] In a seventh aspect, the present application provides a computer-readable storage medium, comprising a computer program, which, when executed on a processor on an electronic device, enables execution of a method in any possible implementation of the first aspect described above.

[0041] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a software system architecture of an electronic device provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the software system architecture interaction flow of an electronic device provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of a flow chart of an image motion estimation method provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a template buffer of a first image frame provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of a template buffer of a second image frame provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of motion vector information between a first image frame and a second image frame provided in an embodiment of the present application;

[0049] Fig. 8A A schematic diagram of generating a target image frame provided in an embodiment of the present application;

[0050] Figure 8B A schematic diagram of generating another target image frame provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of a flow chart of an image motion estimation method provided by another embodiment of the present application;

[0052] Fig.10 A flowchart of an image motion estimation method provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0054] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0055] The following introduces the hardware structure of an electronic device provided in an embodiment of the present application.

[0056] Figure 1 A schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application is shown.

[0057] It should be understood that Figure 1 The electronic device 100 shown is only an example, and the electronic device 100 may have more Figure 1 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0058] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K, etc.

[0059] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0060] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0061] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0062] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0063] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0064] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0065] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0066] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194.

[0067] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0068] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0069] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0070] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0071] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0072] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0073] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0074] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0075] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs.

[0076] NPU is a neural-network (NN) computing processor. It can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.

[0077] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0078] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0079] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0080] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. Speaker 170A, also known as "speaker", is used to convert audio electrical signal into sound signal. Receiver 170B, also known as "earpiece", is used to convert audio electrical signal into sound signal. Microphone 170C, also known as "microphone", "microphone", is used to convert sound signal into electrical signal. Headphone interface 170D is used to connect wired headphones. Gyroscope sensor 180B can be used to determine the motion posture of electronic device 100. Acceleration sensor 180E can detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes). Touch sensor 180K, also known as "touch panel". Touch sensor 180K can be set on display screen 194, and touch sensor 180K and display screen 194 form a touch screen, also known as "touch screen". Touch sensor 180K is used to detect touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 194. The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to the user settings and function control of the electronic device 100. The motor 191 can generate a vibration prompt. The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 195, or pulled out from the SIM card interface 195 to achieve contact and separation with the electronic device 100.

[0081] In the embodiment of the present application, the device type of the electronic device 100 can be a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (extended reality, XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), vehicle-mounted devices or smart city devices, and the like.

[0082] Taking a game application as an example, some concepts related to motion estimation in the embodiments of the present application are introduced below.

[0083] 1. Game frame interpolation.

[0084] During the running of the game application, the electronic device 100 can perform motion estimation on multiple adjacent rendered image frames to obtain motion vector information between multiple adjacent image frames in order to reduce the frame rate of the game application rendering image frames while ensuring the smoothness of the game screen. The electronic device 100 can generate a target image frame based on the motion vector information and multiple adjacent image frames. The electronic device 100 can insert the target image frame into the display queue for display.

[0085] 2. Optical flow motion estimation method.

[0086] Optical flow refers to the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. By using the changes in the pixels in two adjacent image frames, the correspondence between the pixels in the latter image frame and the pixels in the previous image frame can be determined, thereby obtaining the optical flow information between the two adjacent image frames. Among them, the optical flow information of the two adjacent image frames can be used to represent the motion vector information of the object between the two adjacent image frames.

[0087] The optical flow motion estimation method is relatively robust and can be effective in most motion scenes. However, when the object in the frame moves quickly, the result of the optical flow motion estimation will be inaccurate.

[0088] 3. 3D motion field motion estimation method.

[0089] The 3D motion field motion estimation method first determines the motion information of the same object in two adjacent image frames in 3D space, and then projects the motion information of the object in 3D space into a 2D plane to determine the motion vector information of the object between two adjacent image frames.

[0090] The 3D motion field motion estimation method can ensure a relatively accurate motion estimation result even when the object in the image frame moves at a fast speed. However, the 3D motion field motion estimation method has a large amount of calculation when estimating the motion vector of a dynamic object, and the 3D motion field motion estimation method cannot perform motion estimation on semi-transparent objects.

[0091] Therefore, an image motion estimation method is provided in an embodiment of the present application, and the electronic device 100 can intercept the rendering instructions of the application for the image frame, and obtain the relevant resources of the image frame based on the rendering instructions, integrate the optical flow motion estimation method and the 3D motion field motion estimation method, perform motion estimation on two adjacent image frames, and determine the motion vector information between the two adjacent image frames. The electronic device 100 can generate a target image frame based on the motion vector information between the two adjacent image frames and the two adjacent image frames. In this way, the accuracy of the motion estimation result between the two adjacent image frames can be improved.

[0092] The following introduces a software system architecture of an electronic device provided in an embodiment of the present application.

[0093] Figure 2 A schematic diagram of a software system architecture of an electronic device provided in an embodiment of the present application is shown.

[0094] like Figure 2 As shown, the software system architecture of the electronic device 100 may include an application (APP) layer, a native service (native) layer, and a kernel (kernel) layer.

[0095] The application layer includes one or more applications, such as game applications, extended reality (XR) applications, autonomous driving applications, etc. The application can be used to provide image content of an image frame, and call a graphics interface to configure the GPU to render an image frame based on the image content of the image frame.

[0096] The native service layer may include a layer synthesizer (SurfaceFlinger), a motion estimation module, and one or more graphics interfaces, etc. Optionally, the native service layer may also include a frame insertion module and an optical flow algorithm interface. Among them, SurfaceFlinger is used to perform layer synthesis processing on the image frames rendered by the GPU, generate an interface including the image frames, and send the interface including the image frames to the display screen for display. The motion estimation module can be used to intercept the rendering instructions and frame sending instructions issued by the application, and determine the motion vector information between two adjacent image frames based on the relevant resources obtained by intercepting the rendering instructions and frame sending instructions.

[0097] The optical flow algorithm interface may be an algorithm interface provided by the system on chip (SoC) platform of the electronic device 100. The motion estimation module may call the optical flow algorithm interface to obtain the optical flow motion vector information between two adjacent image frames. Among them, the motion estimation module may give the two adjacent image frames to the optical flow algorithm interface when calling the optical flow algorithm interface. After acquiring the two adjacent image frames, the optical flow algorithm interface may calculate the optical flow motion vector information of the two adjacent image frames by the optical flow motion estimation method, and return the optical flow motion vector information of the two adjacent image frames to the motion estimation module.

[0098] The optical flow algorithm interface may be an OpenGL extension interface provided by the SOC platform. For example, the OpenGL extension interface may include a "glTexEstimateMotionQCOM" interface, a "glTexEstimateMotionRegionsQCOM" interface, etc.).

[0099] In a possible implementation manner, the motion estimation module may also determine the optical flow motion vector information of two adjacent image frames by itself using an optical flow motion estimation method.

[0100] Among them, the optical flow motion estimation method may include: pyramid optical flow method, Lucas-Kanade method and neural network-based optical flow method (FlowNet / FlowNet2.0) and other optical flow algorithms.

[0101] Optionally, the motion estimation module may also send the motion vector between two adjacent image frames and the two adjacent image frames to the interpolation module. The interpolation module may generate a target image frame based on the motion vector between two adjacent image frames and the two adjacent image frames.

[0102] The one or more graphics interfaces may include one or more of an open graphics library (OpenGL), an open graphics library for embedded systems (OpenGL ES), and a graphics application programming interface (Vulkan), etc. The graphics interface may be used to configure a GPU to render image frames.

[0103] The kernel layer may include graphics processing unit driver (GPUDriver), etc. Among them, GPU Driver is used to drive GPU to perform upper layer operations (for example, operations of rendering image frames configured by a graphics interface).

[0104] Taking a game application as an example, the following describes in detail the interaction process of each module in the above software system architecture.

[0105] Figure 3 A schematic diagram of the software system architecture interaction process of an electronic device provided in an embodiment of the present application is shown.

[0106] like Figure 3 As shown, the software system architecture interaction process is specifically described by taking a game application as an example. Among them, the game application may include a main thread (Main Thread) and a rendering thread (Render Thread).

[0107] The interaction process of each module can be as follows:

[0108] 1. The main thread of the game application can synchronize the image content to the rendering thread.

[0109] 2. After the rendering thread obtains the image content, it can send rendering instructions to the graphics interface.

[0110] The graphics interface may include any one of OpenGL, OpenGL ES, and Vulkan.

[0111] 3. The graphics interface can send rendering instructions to the motion estimation module.

[0112] 4. The motion estimation module can send rendering instructions to the GPU driver.

[0113] After receiving the rendering instruction, the GPU driver gives the rendering instruction to the GPU to configure the GPU to render an image frame based on the image content, and stores the rendered image frame into the buffer specified by the rendering instruction.

[0114] After rendering an image frame, the GPU can store the image frame in the buffer specified by the rendering instruction.

[0115] 5. After issuing rendering instructions, the rendering thread can send frame instructions to the graphics interface.

[0116] 6. The graphics interface can send frame instructions to the motion estimation module.

[0117] 7. The motion estimation module can forward the frame sending instruction to SurfaceFlinger.

[0118] After SurfaceFlinger obtains the frame sending instruction, it can obtain the image frame buffer information from the frame sending instruction. When the image frame rendered by the GPU is stored in the buffer, SurfaceFlinger can know in time that the image frame has been stored in the buffer through the fence mechanism. Therefore, when the vertical synchronization (vsync) signal arrives, SurfaceFlinger can take the image frame from the image frame buffer for layer synthesis to synthesize the interface including the image frame, and send the interface including the image frame to the display screen for display.

[0119] 8. The motion estimation module can determine the motion vector information between two adjacent image frames based on the relevant resources obtained by intercepting the rendering instruction and the frame sending instruction.

[0120] In a possible implementation, the motion estimation module may identify the type of one or more rendering objects in the image frame based on the rendering instruction of the image frame. The motion estimation module may determine the motion estimation mode corresponding to each of the one or more rendering objects based on the type of the one or more rendering objects. The motion estimation module may determine the motion vector of the one or more rendering objects based on the motion estimation mode corresponding to each of the one or more rendering objects. The motion estimation module may determine the motion vector information of the image frame based on the motion vector of the one or more rendering objects and the pixel position of the one or more rendering objects in the image frame. The motion vector information of the image frame includes the motion vector of each pixel position in the image frame.

[0121] In a possible implementation, the motion estimation module can obtain two adjacent image frames from the buffer of the image frames indicated in the frame sending instruction based on the rendering instruction of the image frame, the transformation matrix and depth information of the two adjacent image frames, and the frame sending instruction. The motion estimation module can determine the motion vector information between the two adjacent image frames by fusing the optical flow motion estimation method and the 3D motion field motion estimation method based on the two adjacent image frames and the transformation matrix and depth information of the two adjacent image frames.

[0122] When the motion vector information between two adjacent image frames determined by the motion estimation module is used for frame interpolation, the following steps may be included:

[0123] 9. The motion estimation module provides the motion vector information between two adjacent image frames and the two adjacent image frames to the interpolation module.

[0124] 10. The interpolation module generates a target image frame based on the motion vector information between two adjacent image frames and the two adjacent image frames.

[0125] After acquiring the rendered image frame, the frame insertion module may determine the target image frame based on two adjacent image frames and motion vector information of the two adjacent image frames.

[0126] The frame insertion module may insert a target image frame between two adjacent image frames, or may generate a target image frame after two adjacent image frames.

[0127] For example, two adjacent image frames may include a first image frame and a second image frame, and the rendering time of the second image frame is later than the rendering time of the first image frame. When the interpolation module determines to insert a target image frame between the first image frame and the second image frame, the interpolation module may determine the target image frame based on the first image frame and the second image frame, and the motion vector information between the first image frame and the second image frame. The interpolation module may insert the target image frame between the first image frame and the second image frame and send it to SurfaceFlinger for synthesis and display.

[0128] When the interpolation module determines to insert the target image frame after the second image frame, the interpolation module can determine the target image frame based on the second image frame and the motion vector information between the first image frame and the second image frame. The interpolation module can place the target image frame after the second image frame and send it to SurfaceFlinger for synthesis and display.

[0129] The interpolation module can store the target image frame into the buffer of the target image frame.

[0130] 11. The interpolation module can send the buffer address of the target image frame to SurfaceFlinger.

[0131] After obtaining the buffer of the target image frame, SurfaceFlinger can wait until the next vsync signal arrives, take out the target image frame from the buffer of the target image frame for layer synthesis, so as to synthesize the interface including the target image frame, and send the interface including the target image frame to the display screen for display.

[0132] The following describes motion estimation provided in an embodiment of the present application in conjunction with application scenarios.

[0133] Application scenario 1:

[0134] Since some game applications support functions that require motion estimation, such as motion blur, when the viewing angle of the image frame is in a game scene such as rotation, the game application will render the characters or background objects in the image frame into the effect of being captured by the camera under high-speed motion, so that the objects in the image frame look blurry. In order to achieve the effect of motion blur, when rendering the image frame, the game application can save the object's motion field motion vector by projecting the motion information of the object in the image frame in three-dimensional space to a two-dimensional image. The game application can send the motion field motion vector information in the image frame to the graphics interface through a rendering instruction. Among them, the motion field motion vector information in the image frame includes the motion field motion vector of the pixel position where the opaque object is located, but the motion field motion vector information in the image frame does not include the motion field motion vector of the pixel position where the semi-transparent object (for example, shadows, skill special effects, etc.) is located. Therefore, the motion estimation module can intercept the rendering instructions of the game application for each rendered object in the image frame, thereby obtaining the motion field motion vector of the pixel position where each rendered object is located, and using the motion field motion vector of the pixel position where the opaque object is located as the motion vector of the pixel position where the opaque object is located. For semi-transparent objects, the motion estimation module can calculate the optical flow information between two adjacent image frames, and then determine the optical flow motion vector information between the two adjacent image frames from the optical flow information, and use the optical flow motion vector of the pixel position of the semi-transparent object as the motion vector of the pixel position of the opaque object.

[0135] Figure 4 A flow chart of an image motion estimation method provided in an embodiment of the present application is shown.

[0136] like Figure 4 As shown, the image motion estimation method may include the following steps:

[0137] S401. A motion estimation module may intercept, from a graphics interface, a rendering instruction of a game application for rendering an object in two adjacent image frames.

[0138] The two adjacent image frames include a first image frame and a second image frame, and the rendering time of the second image frame is later than the rendering time of the first image frame.

[0139] The game application may first send the rendering instruction of the first image frame to the graphics interface. After sending the rendering instruction of the first image frame, the game application may send the frame sending instruction of the first image frame to SurfaceFlinger through the graphics interface. After sending the frame sending instruction of the first image frame to SurfaceFlinger through the graphics interface, the game application may send the rendering instruction of the second image frame to the graphics interface. After sending the rendering instruction of the second image frame, the game application may send the frame sending instruction of the second image frame to SurfaceFlinger through the graphics interface.

[0140] The rendering instructions may include multiple instructions for configuring the graphics interface to call the GPU to render the image frame. When the image frame is rendered, it is rendered in different regions according to the objects in the image frame. Therefore, the rendering instructions may indicate a label list of multiple objects in the image frame and the pixel position of each object in the image frame.

[0141] S402: The motion estimation module may determine whether the rendering instruction contains a keyword of a semi-transparent object.

[0142] For example, the motion estimation module can intercept the "glLabelObjectEXT" instruction in the rendering instruction, and the motion estimation module can determine whether there is a keyword of a semi-transparent object in the label carried by the "glLabelObjectEXT" instruction. If yes, the motion estimation module can determine that the rendering object corresponding to the "glLabelObjectEXT" instruction is a semi-transparent object. If not, the motion estimation module can determine that the rendering object corresponding to the "glLabelObjectEXT" instruction is an opaque object.

[0143] S403: If the rendering instruction includes a keyword of a semi-transparent object, the motion estimation module may determine that the rendering object is a semi-transparent object, and record pixel positions of the semi-transparent object in the first image frame and the second image frame respectively.

[0144] S404: If the rendering instruction does not include a keyword of a semi-transparent object, the motion estimation module may determine that the rendering object is an opaque object, and record pixel positions of the opaque object in the first image frame and the second image frame respectively.

[0145] In the process of rendering the first image frame, the motion estimation module can identify and record the pixel position of the semi-transparent object in the first image frame by intercepting the rendering instruction of the first image frame. In the process of rendering the second image frame, the motion estimation module can identify and record the pixel position of the semi-transparent object in the second image frame by intercepting the rendering instruction of the second image frame.

[0146] Specifically, the motion estimation module may enable a stencil test during the rendering of the first image frame, and create a stencil buffer A, wherein the [i, j]th stencil value in the stencil buffer A corresponds to the [i, j]th pixel position in the first image frame. i represents the row where the pixel position is located, and j represents the column where the pixel position is located. i and j are positive integers. The motion estimation module may record the stencil value of the pixel position corresponding to the semi-transparent object in the first image frame in the stencil buffer A as St rans, the template value of the pixel position corresponding to the opaque object in the first image frame is recorded as St opaq For example, St rans The value can be 0, St opaq The value can be 1.

[0147] For example, Figure 5 As shown, the pixel size of the first image frame can be 9*9, that is, there can be 9 rows*9 columns in the first image frame, with a total of 81 pixels. Therefore, the template buffer A can also include 9 rows*9 columns, with a total of 81 template values. Among them, objects 1 and 2 in the first image frame can be opaque objects, and objects at other pixel positions in the first image frame can be semi-transparent objects. Since the pixel positions of object 1 in the first image frame include the [2, 1]th pixel, the [2, 2]th pixel, the [3, 1]th pixel and the [3, 2]th pixel. The pixel positions of object 2 in the first image frame include the [7, 1]th pixel, the [7, 2]th pixel, the [7, 1]th pixel and the [7, 2]th pixel. Therefore, the template values ​​[2, 1]th, [2, 2]th, [3, 1]th, and [3, 2]th, as well as the template values ​​[7, 1]th, [7, 2]th, [7, 1]th, and [7, 2]th in the template buffer A can all be recorded as 1. Template values ​​at other locations in the template buffer A can be recorded as 0.

[0148] The motion estimation module may start a stencil test during rendering of the second image frame, and create a stencil buffer B, wherein the [i, j]th stencil value in the stencil buffer B corresponds to the [i, j]th pixel position in the second image frame, where i and j are positive integers. The motion estimation module may record the stencil value of the pixel position corresponding to the semi-transparent object in the second image frame in the stencil buffer B as St rans , the template value of the pixel position corresponding to the opaque object in the second image frame is recorded as St opaq For example, St rans The value of St can be 0. opaq The value of can be 1.

[0149] For example, Figure 6As shown, the pixel size of the second image frame can be 9*9, that is, there can be 9 rows*9 columns in the second image frame, with a total of 81 pixels. Therefore, the template buffer B can also include 9 rows*9 columns, with a total of 81 template values. Among them, objects 1 and 2 in the second image frame can be opaque objects, and objects at other pixel positions in the second image frame can be semi-transparent objects. Since the pixel positions of object 1 in the second image frame include the [2,3]th pixel, the [2,4]th pixel, the [3,3]th pixel and the [3,4]th pixel. The pixel positions of object 2 in the second image frame include the [5,1]th pixel, the [5,2]th pixel, the [6,1]th pixel and the [6,2]th pixel. Therefore, the template values ​​[2,3]th, [2,4]th, [3,3]th, and [3,4]th, as well as the template values ​​[5,1]th, [5,2]th, [6,1]th, and [6,2]th in the template buffer B can all be recorded as 1. Template values ​​at other positions in the template buffer B can be recorded as 0.

[0150] S405 , the motion estimation module may determine the motion vector of the pixel position where the semi-transparent object is located between the first image frame and the second image frame by using an optical flow motion estimation method.

[0151] Among them, since the motion estimation module records the pixel positions of the semi-transparent objects in the first image frame and the second image frame respectively through the template buffer A and the template buffer B. Therefore, the motion estimation module can obtain the buffer of the first image frame and the buffer of the second image frame by intercepting the rendering instruction at the graphics interface. The motion estimation module can take out the rendered first image frame from the buffer of the first image frame, and take out the rendered second image frame from the buffer of the second image frame. The motion estimation module can determine the pixel position of the semi-transparent object in the first image frame based on the template buffer A and the first image frame, and determine the pixel position of the semi-transparent object in the second image frame based on the template buffer B and the second image frame.

[0152] The motion estimation module can determine the optical flow information of the semi-transparent object between the first image frame and the second image frame through the optical flow motion estimation method based on the first image frame, the second image frame and the pixel positions of the semi-transparent object in the first image frame and the second image frame respectively.

[0153] The motion estimation module may determine a motion vector of a pixel position where the semi-transparent object is located between the first image frame and the second image frame based on optical flow information of the semi-transparent object between the first image frame and the second image frame.

[0154] Among them, the motion estimation module can determine the optical flow information of the semi-transparent object between the first image frame and the second image frame through optical flow motion estimation methods such as the pyramid optical flow method, the Lucas-Kanade method, and the neural network-based optical flow method (FlowNet / FlowNet2.0).

[0155] In a possible implementation, the motion estimation module may provide the first image frame and the second image frame to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface may calculate the optical flow motion vector information of two adjacent image frames by an optical flow motion estimation method, and return the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module. The motion estimation module may determine the motion vector of the pixel position of the semi-transparent object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel positions of the semi-transparent object in the first image frame and the second image frame, respectively.

[0156] In a possible implementation, the motion estimation module may provide the pixel positions of the first image frame, the second image frame, and the semi-transparent object in the first image frame and the second image frame, respectively, to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface may determine the motion vector of the pixel position of the semi-transparent object between the first image frame and the second image frame based on the pixel positions of the first image frame, the second image frame, and the semi-transparent object in the first image frame and the second image frame, respectively. The optical flow algorithm interface may return the motion vector of the pixel position of the semi-transparent object between the first image frame and the second image frame to the motion estimation module.

[0157] S406: The motion estimation module may obtain native motion vector information of the game application from the graphic interface, and determine the motion vector of the pixel position where the opaque object is located between the first image frame and the second image frame from the native motion vector information.

[0158] For example, the motion estimation module can obtain the native motion vector information of the game application from the graphics interface through the "glGetUniformLocation" instruction.

[0159] The motion estimation module can determine the motion vector of the pixel position of the opaque object between the first image frame and the second image frame from the native motion vector information based on the pixel position of the opaque object in the first image frame and the pixel position of the opaque object in the second image frame.

[0160] Optionally, after the motion estimation module determines the motion vector information between the first image frame and the second image frame, step S407 may be performed.

[0161] S407 . The frame insertion module may generate a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame.

[0162] Among them, after obtaining the motion vector information between the first image frame and the second image frame sent by the motion estimation module, the interpolation module can insert the target image frame between the first image frame and the second image frame, or insert the target image frame after the first image frame and the second image frame.

[0163] When the interpolation module determines to insert a target image frame between the first image frame and the second image frame, the interpolation module may generate the target image frame based on the first image frame and the second image frame, and motion vector information between the first image frame and the second image frame.

[0164] For example, Figure 7 As shown, the motion vector information between the first image frame and the second image frame may include the motion vector of object 1 and the motion vector of object 2. The pixel positions of object 1 in the first image frame include the [2, 1]th pixel, the [2, 2]th pixel, the [3, 1]th pixel, and the [3, 2]th pixel. The pixel positions of object 1 in the second image frame include the [2, 3]th pixel, the [2, 4]th pixel, the [3, 3]th pixel, and the [3, 4]th pixel. The motion vector of object 1 indicates that object 1 has moved 2 pixel positions to the right.

[0165] The pixel positions of object 2 in the first image frame include the [7, 1]th pixel, the [7, 2]th pixel, the [7, 1]th pixel, and the [7, 2]th pixel. The pixel positions of object 2 in the second image frame include the [5, 1]th pixel, the [5, 2]th pixel, the [6, 1]th pixel, and the [6, 2]th pixel. The motion vector of object 2 indicates that object 2 has moved 2 pixel positions upward.

[0166] like Fig. 8A As shown, when the interpolation module determines to insert the target image frame between the first image frame and the second image frame, from the first image frame to the target image frame, object 1 needs to move 1 pixel position to the right, and object 2 needs to move 1 pixel position upward. Therefore, the pixel position of object 1 in the target image frame includes the [2, 2]th pixel, the [2, 3]th pixel, the [3, 2]th pixel, and the [3, 3]th pixel. The pixel position of object 2 in the target image frame includes the [6, 1]th pixel, the [6, 2]th pixel, the [7, 1]th pixel, and the [7, 2]th pixel.

[0167] When the interpolation module determines to generate the target image frame after the second image frame, the interpolation module may generate the target image frame based on the second image frame and motion vector information between the first image frame and the second image frame.

[0168] like Figure 8B As shown, when the interpolation module determines to generate a target image frame after the second image frame, from the second image frame to the target image frame, object 1 needs to move 2 pixel positions to the right, and object 2 needs to move 2 pixel positions upward. Therefore, the pixel positions of object 1 in the target image frame include the [2, 5]th pixel, the [2, 6]th pixel, the [3, 5]th pixel, and the [3, 6]th pixel. The pixel positions of object 2 in the target image frame include the [3, 1]th pixel, the [3, 2]th pixel, the [4, 1]th pixel, and the [4, 2]th pixel.

[0169] The above examples are only used to explain the present application and should not be construed as limiting.

[0170] An image motion estimation method is provided in an embodiment of the present application. By intercepting the rendering instructions of each rendering object in the image frame, different motion estimation methods can be used for different types of rendering objects. In this way, different motion estimation methods are applicable to different rendering objects. For example, the effect of the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. However, when the object in the picture frame moves faster, the result of the optical flow motion estimation will have errors. The 3D motion field motion estimation method can also ensure more accurate motion estimation results when the object in the image frame moves faster. However, the 3D motion field motion estimation method cannot perform motion estimation on translucent objects. The native motion field vector information of the application is determined by the 3D motion field motion estimation method. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can improve the accuracy of the motion estimation results between two adjacent image frames.

[0171] For example, an optical flow motion estimation method is used to determine a motion vector of a first object between two adjacent image frames, and a motion vector of the first object between two adjacent image frames is obtained from the native motion vector of the application. In an embodiment of the present application, the first object may be a semi-transparent object, and the second object may be an opaque object. In other embodiments, the first object and the second object may be divided into other types.

[0172] Application scenario 2:

[0173] Since the motion estimation module cannot directly obtain the motion vector of the playing field from the rendering instructions of the game application to each rendering object in the image frame, the motion estimation module can obtain the transformation matrix and depth information of two adjacent image frames by intercepting the rendering instructions of the game application to each rendering object in the image frame. The motion estimation module can determine the motion vector of the playing field at the pixel position where the static object is located between the two adjacent image frames through the transformation matrix and depth information of the two adjacent image frames. The motion estimation module can calculate the optical flow information between the two adjacent image frames, and determine the motion vector of the pixel position where the dynamic object is located between the two adjacent image frames through the optical flow information.

[0174] Fig. 9 A schematic flow chart of an image motion estimation method provided in another embodiment of the present application is shown.

[0175] like Fig. 9 As shown, the image motion estimation method may include the following steps:

[0176] S901. A motion estimation module may intercept, from a graphics interface, rendering instructions for rendering objects in two adjacent image frames of a game application.

[0177] The two adjacent image frames include a first image frame and a second image frame, and the rendering time of the second image frame is later than the rendering time of the first image frame.

[0178] For details, please refer to the above Figure 4 Step S401 in the illustrated embodiment will not be described in detail here.

[0179] S902: The motion estimation module may determine whether the rendering instruction contains a keyword of a dynamic object.

[0180] For example, the motion estimation module can intercept the "glLabelObjectEXT" instruction in the rendering instruction, and the motion estimation module can determine whether the label carried by the "glLabelObjectEXT" instruction contains a keyword of a dynamic object (for example, a person, a monster, a shadow, etc.). If yes, the motion estimation module can determine that the rendering object corresponding to the "glLabelObjectEXT" instruction is a dynamic object. If no, the motion estimation module can determine that the rendering object corresponding to the "glLabelObjectEXT" instruction is a static object.

[0181] S903: If the rendering instruction contains a keyword of a dynamic object, the motion estimation module may determine that the rendering object is a dynamic object, and record pixel positions of the dynamic object in the first image frame and the second image frame respectively.

[0182] S904: If the rendering instruction does not include a keyword of a dynamic object, the motion estimation module may determine that the rendering object is a static object, and record pixel positions of the static object in the first image frame and the second image frame respectively.

[0183] In the process of rendering the first image frame, the motion estimation module can identify and record the pixel position of the dynamic object in the first image frame by intercepting the rendering instruction of the first image frame. In the process of rendering the second image frame, the motion estimation module can identify and record the pixel position of the dynamic object in the second image frame by intercepting the rendering instruction of the second image frame.

[0184] Specifically, the motion estimation module may enable a stencil test during the rendering of the first image frame, and create a stencil buffer C, wherein the [i, j]th stencil value in the stencil buffer C corresponds to the [i, j]th pixel position in the first image frame. i represents the row where the pixel position is located, and j represents the column where the pixel position is located. i and j are positive integers. The motion estimation module may record the stencil value of the pixel position corresponding to the dynamic object in the first image frame in the stencil buffer C as St dyna , the template value of the pixel position corresponding to the opaque object in the first image frame is recorded as St sta For example, St dyna The value can be 0, St sta The value can be 1.

[0185] In one example, the template value in the template buffer C can refer to the above Figure 5 The stencil value in the stencil buffer A in the illustrated embodiment.

[0186] The motion estimation module may start a stencil test and create a stencil buffer D during the rendering of the second image frame. The stencil value [i, j] in the stencil buffer D corresponds to the pixel position [i, j] in the second image frame. i represents the row where the pixel position is located, and j represents the column where the pixel position is located. i and j are positive integers. The motion estimation module may record the stencil value of the pixel position corresponding to the dynamic object in the second image frame in the stencil buffer D as St dyna , the template value of the pixel position corresponding to the opaque object in the second image frame is recorded as St sta For example, St dyna The value can be 0, St sta The value can be 1.

[0187] In one example, the template value in the template buffer D can refer to the above Figure 6 The stencil value in the stencil buffer B in the embodiment shown.

[0188] S905 . The motion estimation module may determine the motion vector of the pixel position where the dynamic object is located between the first image frame and the second image frame by using an optical flow motion estimation method.

[0189] Among them, since the motion estimation module records the pixel positions of the dynamic objects in the first image frame and the second image frame respectively through the template buffer C and the template buffer D. Therefore, the motion estimation module can obtain the buffer of the first image frame and the buffer of the second image frame by intercepting the rendering instruction at the graphics interface. The motion estimation module can take out the rendered first image frame from the buffer of the first image frame, and take out the rendered second image frame from the buffer of the second image frame. The motion estimation module can determine the pixel position of the dynamic object in the first image frame based on the template buffer C and the first image frame, and determine the pixel position of the dynamic object in the second image frame based on the template buffer D and the second image frame.

[0190] The motion estimation module can determine the optical flow information of the dynamic object between the first image frame and the second image frame through the optical flow motion estimation method based on the first image frame, the second image frame and the pixel positions of the dynamic object in the first image frame and the second image frame respectively.

[0191] The motion estimation module may determine a motion vector of a pixel position where the dynamic object is located between the first image frame and the second image frame based on optical flow information of the dynamic object between the first image frame and the second image frame.

[0192] Among them, the motion estimation module can determine the optical flow information of the dynamic object between the first image frame and the second image frame through optical flow motion estimation methods such as the pyramid optical flow method, the Lucas-Kanade method, and the neural network-based optical flow method (FlowNet / FlowNet2.0).

[0193] In a possible implementation, the motion estimation module can provide the first image frame and the second image frame to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface can calculate the optical flow motion vector information of two adjacent image frames by the optical flow motion estimation method, and return the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module. The motion estimation module can determine the motion vector of the pixel position of the dynamic object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel positions of the dynamic object in the first image frame and the second image frame respectively.

[0194] In a possible implementation, the motion estimation module may provide the pixel positions of the first image frame, the second image frame, and the dynamic object in the first image frame and the second image frame, respectively, to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface may determine the motion vector of the pixel position of the dynamic object between the first image frame and the second image frame based on the pixel positions of the first image frame, the second image frame, and the dynamic object in the first image frame and the second image frame, respectively. The optical flow algorithm interface may return the motion vector of the pixel position of the dynamic object between the first image frame and the second image frame to the motion estimation module.

[0195] S906: The motion estimation module may obtain the transformation matrix and depth information of the first image frame, and the transformation matrix and depth information of the second image frame from the graphic interface.

[0196] The motion estimation module can obtain the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame by intercepting relevant instructions such as the "glBufferSubData" instruction or the "glGetUniformLocation" instruction in the rendering instruction in the graphics interface.

[0197] The transformation matrix of the first image frame is used to represent the transformation relationship of the first image frame from the viewing angle coordinate system of the first image frame to the world coordinate system. The transformation matrix of the second image frame is used to represent the transformation relationship of the second image frame from the viewing angle coordinate system of the first image frame to the world coordinate system.

[0198] S907. The motion estimation module can determine the motion vector of the pixel position of the static object between the first image frame and the second image frame through a 3D motion field motion estimation method based on the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame.

[0199] Since the motion vector of the static object in the first image frame is generated by the perspective transformation between the first image frame and the second image frame, the motion estimation module can reproject the depth information of the second image frame to the perspective coordinates of the first image frame to obtain the depth information of the second image frame in the perspective coordinate system of the first image frame. The motion estimation module can determine the motion vector of the static object between the first image frame and the second image frame based on the depth information of the second image frame in the perspective coordinate system of the first image frame and the depth information of the first image frame.

[0200] The calculation formula of the motion vector of the static object between the first image frame and the second image frame may be as follows:

[0201]

[0202] In the above formula (1), v 3D-S is the motion vector of the pixel position of the static object between the first image frame and the second image frame, VP n-1 is the transformation matrix of the first image frame, is the transposed matrix of the transformation matrix of the second image frame. cur-S is the depth value of the pixel position where the static object is located in the second image frame. pre-S is the depth value of the pixel position where the static object is located in the first image frame.

[0203] Optionally, after the motion estimation module determines the motion vector information between the first image frame and the second image frame, step S908 may be executed.

[0204] S908: The frame insertion module may generate a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame.

[0205] For details, please refer to the above Figure 4 Step S407 in the illustrated embodiment will not be described in detail here.

[0206] Through an image motion estimation method provided by the present application, different motion estimation methods can be used for different types of rendering objects by intercepting the rendering instructions of the game application for each rendering object in the image frame. In this way, different motion estimation methods are applicable to different rendering objects. For example, the effect of the optical flow motion estimation method is relatively robust and can be effective in most motion scenes. The 3D motion field motion estimation method can also ensure more accurate motion estimation results when the object in the image frame moves faster. However, the motion vector of the dynamic object is relatively cumbersome to calculate using the 3D motion field estimation method, and the calculation time is relatively long. Compared with using the same motion estimation method for different rendering objects, using different rendering objects for different types of rendering objects can quickly determine the motion estimation results between two adjacent image frames.

[0207] For example, an optical flow motion estimation method is used to determine the motion vector of the first object between two adjacent image frames, and a 3D motion field motion estimation method is used to determine the motion vector of the first object between two adjacent image frames. In an embodiment of the present application, the first object may be a dynamic object, and the second object may be a static object. In other embodiments, the first object and the second object may be divided into other types.

[0208] Application scenario 3:

[0209] Since the motion estimation module cannot directly obtain the motion vector of the motion field from the rendering instructions of the game application for each rendered object in the image frame. In addition, some game applications define different labels for each rendered object in the image frame, which may cause the motion estimation module to be unable to identify the object type in the image frame. Therefore, the motion estimation module can determine the motion vector information between two adjacent image frames by combining the transformation matrix and depth information of two adjacent image frames and the optical flow motion estimation method and the 3D motion field motion estimation method.

[0210] Fig.10 A schematic flow chart of an image motion estimation method provided in another embodiment of the present application is shown.

[0211] like Fig.10 As shown, the image motion estimation method may include the following steps:

[0212] S1001. A motion estimation module may intercept rendering instructions of two adjacent image frames of a game application from a graphics interface to obtain two adjacent image frames and transformation matrices and depth information of the two adjacent image frames.

[0213] The motion estimation module can obtain the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame by intercepting relevant instructions such as the "glBufferSubData" instruction or the "glGetUniformLocation" instruction in the rendering instruction in the graphics interface.

[0214] The transformation matrix of the first image frame is used to represent the transformation relationship of the first image frame from the viewing angle coordinate system of the first image frame to the world coordinate system. The transformation matrix of the second image frame is used to represent the transformation relationship of the second image frame from the viewing angle coordinate system of the first image frame to the world coordinate system.

[0215] The depth information of the first image frame includes the depth value of each pixel position in the first image frame. The depth information of the second image frame includes the depth value of each pixel position in the second image frame.

[0216] S1002: The motion estimation module may determine optical flow motion vector information between the first image frame and the second image frame by using an optical flow motion estimation method.

[0217] The motion estimation module can determine the optical flow motion vector information between the first image frame and the second image frame through an optical flow motion estimation method such as a pyramid optical flow method, a Lucas-Kanade method, and an optical flow method based on a neural network (FlowNet / FlowNet2.0).

[0218] In a possible implementation, the motion estimation module can provide the first image frame and the second image frame to the optical flow algorithm interface provided by the SoC. The optical flow algorithm interface can calculate the optical flow motion vector information of two adjacent image frames by an optical flow motion estimation method, and return the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module.

[0219] The optical flow motion vector information between the first image frame and the second image frame may include the optical flow motion vector of each pixel position in the second image frame between the first image frame and the second image frame.

[0220] S1003. The motion estimation module may determine the motion field motion vector information between the first image frame and the second image frame through a 3D motion field motion estimation method based on the transformation matrix and depth information of the first image frame and the transformation matrix and depth information of the second image frame.

[0221] The motion estimation module may reproject the depth information of the second image frame to the viewing angle coordinates of the first image frame to obtain the depth information of the second image frame in the viewing angle coordinate system of the first image frame. The motion estimation module may determine the motion field motion vector information between the first image frame and the second image frame based on the depth information of the second image frame in the viewing angle coordinate system of the first image frame and the depth information of the first image frame.

[0222] The motion field motion vector information between the first image frame and the second image frame may include the motion field motion vector of each pixel position in the second image frame between the first image frame and the second image frame.

[0223] The calculation formula of the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame may be as follows:

[0224]

[0225] In the above formula (2), v 3D is the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame, VP n-1 is the transformation matrix of the first image frame, is the transposed matrix of the transformation matrix of the second image frame. cur is the depth value of the first pixel position in the second image frame. pre The depth value of the first pixel position in the second image frame reprojected to the second pixel position in the first image frame.

[0226] The first pixel position may be any pixel position in the second image frame. Therefore, the motion vector of the motion field between the first image frame and the second image frame at any pixel position in the second image frame may be determined based on the above formula (2).

[0227] S1004: The motion estimation module may determine whether a difference between an optical flow motion vector at a first pixel position in the second image frame and a motion field motion vector at the first pixel position is less than a specified threshold 1.

[0228] If the difference between the optical flow motion vector at the first pixel position in the second image frame and the motion field motion vector at the first pixel position is less than the specified threshold 1, step S1005 is executed.

[0229] If the difference between the optical flow motion vector at the first pixel position in the second image frame and the motion field motion vector at the first pixel position is greater than or equal to the specified threshold 1, steps S1006 to S1009 are executed.

[0230] S1005. The motion estimation module may use an average value of the optical flow motion vector and the motion field motion vector at the first pixel position as the motion vector at the first pixel position.

[0231] If the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is less than a specified threshold value 1, it can be considered that at the first pixel position, the optical flow motion vector determined by the optical flow motion estimation method and the motion field motion vector determined by the 3D motion field motion estimation method are consistent. Therefore, the average value of the optical flow motion vector at the first pixel position and the motion field motion vector can be used as the motion vector at the first pixel position.

[0232] That is:

[0233] v out =(v 3D +v 2D ) / 2 formula (3)

[0234] In the above formula (3), v out is the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame, v 3D is the motion vector of the motion field between the first image frame and the second image frame at the first pixel position in the second image frame, v 2D is the optical flow motion vector of the first pixel position in the second image frame between the first image frame and the second image frame.

[0235] S1006: The motion estimation module may obtain a depth value of the first pixel position and depth values ​​of surrounding pixel positions around the first pixel position in the second image frame.

[0236] If the difference between the optical flow motion vector at the first pixel position and the motion field motion vector at the first pixel position is greater than or equal to the specified threshold 1, it can be considered that at the first pixel position, the optical flow motion vector determined by the optical flow motion estimation method and the motion field motion vector determined by the 3D motion field motion estimation method are greatly different. Therefore, it can be determined whether the depth value at the first pixel position differs from the depth values ​​at the surrounding pixel positions by less than the specified threshold 2, thereby determining whether the motion vector at the first pixel position is the optical flow motion vector or the motion field motion vector.

[0237] S1007: The motion estimation module may determine whether the difference between the depth value of the first pixel position and the depth values ​​of the surrounding pixel positions is less than a specified threshold 2.

[0238] If the difference between the depth value of the first pixel position and the depth values ​​of the surrounding pixel positions is less than the specified threshold 2, step S1008 is executed.

[0239] If the depth value of the first pixel position differs from the depth values ​​of the surrounding pixel positions by a value greater than or equal to the specified threshold 2, step S1009 is executed.

[0240] The first pixel position has a plurality of surrounding pixel positions. For example, there may be 8 surrounding pixel positions around the first pixel position.

[0241] In a possible implementation, when determining whether the depth value of the first pixel position differs from the depth values ​​of the surrounding pixels by less than a specified threshold 2, the difference between the depth values ​​of the first pixel position and the plurality of surrounding pixel positions may be calculated. If the average value of the difference between the depth values ​​of the first pixel position and the plurality of surrounding pixel positions is less than the specified threshold 2, it may be determined that the depth value of the first pixel position differs from the depth values ​​of the surrounding pixels by less than the specified threshold 2.

[0242] In a possible implementation, when determining whether the difference between the depth value of the first pixel position and the depth values ​​of the surrounding pixels is less than a specified threshold 2, the difference between the depth values ​​of the first pixel position and the plurality of surrounding pixel positions may be calculated. If the difference between the depth values ​​of the first pixel position and the plurality of surrounding pixel positions is less than the specified threshold 2, it may be determined that the difference between the depth value of the first pixel position and the depth values ​​of the surrounding pixels is less than the specified threshold 2.

[0243] S1008. The motion estimation module may use the optical flow motion vector at the first pixel position as the motion vector at the first pixel position.

[0244] S1009. The motion estimation module may use the motion field motion vector at the first pixel position as the motion vector at the first pixel position.

[0245] Since the depth values ​​of each pixel position in a semi-transparent object are not very different, if the depth value difference between the first pixel position and the surrounding pixel positions is small, it means that the area where the first pixel position is located is likely to be the background (that is, the semi-transparent object). For semi-transparent objects, the optical flow motion estimation method can be used to obtain an accurate motion vector. Therefore, if the depth value of the first pixel position differs from the depth value of the surrounding pixels by less than a specified threshold of 2, the optical flow motion vector v of the first pixel position can be used as the background. 2D , as the motion vector v at the first pixel position out , that is, v out =v 2D .

[0246] Since the depth values ​​of each pixel position in an opaque object are quite different, if the depth value of the first pixel position is significantly different from that of the surrounding pixel positions, it means that the area where the first pixel position is located is likely to be an opaque object. For opaque objects, the 3D motion field motion estimation method can be used to obtain an accurate motion vector. Therefore, if the depth value of the first pixel position differs from the depth value of the surrounding pixels by more than a specified threshold of 2, the motion field motion vector v of the first pixel position can be used as the motion vector. 3D . As the motion vector v at the first pixel position out That is, v out =v 2D .

[0247] In some embodiments, since the optical flow motion vectors of each pixel position in a semi-transparent object are not very different, if the optical flow motion vectors of the first pixel position are slightly different from those of the surrounding pixel positions, it means that the area where the first pixel position is located is likely to be the background (that is, the semi-transparent object). For semi-transparent objects, an accurate motion vector can be obtained by using the optical flow motion estimation method. Therefore, if the difference between the optical flow motion vector of the first pixel position and the optical flow motion vectors of the surrounding pixels is less than a specified threshold of 3, the optical flow motion vector v of the first pixel position can be used as the background. 2D , as the motion vector v at the first pixel position out That is, v out =v 2D .

[0248] Since the optical flow motion vectors of each pixel position in an opaque object are quite different, if the optical flow motion vectors of the first pixel position are significantly different from those of the surrounding pixel positions, it means that the area where the first pixel position is located is likely to be an opaque object. For opaque objects, the 3D motion field motion estimation method can be used to obtain an accurate motion vector. Therefore, if the difference between the optical flow motion vector of the first pixel position and the optical flow motion vector of the surrounding pixels is greater than or equal to the specified threshold 3, the motion field motion vector v of the first pixel position can be used as the motion vector. 3D , as the motion vector v at the first pixel positionout That is, v out =v 2D .

[0249] In other embodiments, the depth value difference and the optical flow motion vector difference of the first pixel position can be comprehensively considered to determine whether the first pixel position belongs to a semi-transparent object or an opaque object, thereby determining whether the motion vector of the first pixel position is an optical flow motion vector or a motion field motion vector.

[0250] If the depth value of the first pixel position differs from the depth value of the surrounding pixels by less than a specified threshold of 2, and the optical flow motion vector of the first pixel position differs from the optical flow motion vector of the surrounding pixels by less than a specified threshold of 3, the optical flow motion vector v at the first pixel position can be 2D , as the motion vector v at the first pixel position out That is, v out =v 2D .

[0251] If the depth value of the first pixel position differs from the depth value of the surrounding pixels by more than or equal to the specified threshold 2, or the optical flow motion vector of the first pixel position differs from the optical flow motion vector of the surrounding pixels by more than or equal to the specified threshold 3, the optical flow motion vector v at the first pixel position may be 2D , as the motion vector v at the first pixel position out That is, v out =v 2D .

[0252] Optionally, after the motion estimation module determines the motion vector information between the first image frame and the second image frame, step S1010 may be performed.

[0253] S1010: The interpolation module may generate a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame.

[0254] For details, please refer to the above Figure 4 Step S407 in the illustrated embodiment will not be described in detail here.

[0255] Therefore, an image motion estimation method is provided in an embodiment of the present application, which can perform motion estimation on two adjacent image frames by integrating the transformation matrix and depth information of two adjacent image frames with the optical flow motion estimation method and the 3D motion field motion estimation method, and determine the motion vector information between the two adjacent image frames. In this way, the accuracy of the motion estimation result can be improved.

[0256] In the above embodiment of the present application, the motion estimation module may first determine whether the native motion vector information of the first application can be obtained from the graphics interface, and if the native motion vector information of the first application can be obtained from the graphics interface, the above Figure 4 In the method embodiment shown, if the native motion vector information of the first application cannot be obtained from the graphics interface, the above Fig. 9 The method embodiment is shown.

[0257] Optionally, the motion estimation module may first determine whether the native motion vector information of the first application and the keyword of the rendering object can be obtained from the graphics interface. If the native motion vector information of the first application and the keyword of the rendering object can be obtained from the graphics interface, the above steps are executed. Figure 4 If the keyword of the rendering object can be obtained, but the native motion vector information of the first application cannot be obtained from the graphics interface, the above Fig. 9 If the keyword of the rendering object cannot be obtained, and the native motion vector information of the first application cannot be obtained from the graphics interface, the above Fig.10 The method embodiment is shown.

[0258] In an embodiment of the present application, the motion vector information between two adjacent image frames determined by the above-mentioned image motion estimation method can be used in addition to image frame interpolation in game applications, and can also be used in other scenarios, which is not limited in the embodiment of the present application. For example, image frame interpolation in XR applications, and moving object prediction in autonomous driving. Among them, in XR applications or autonomous driving scenarios, two adjacent image frames can be obtained through a camera, and the depth information of the two adjacent image frames can be collected by a depth sensor, and then the motion field information and optical flow information in the two adjacent image frames can be determined. For another example, the motion vector information between two adjacent image frames determined by the above-mentioned image motion estimation method can also be used to perform motion estimation in game applications that do not have their own motion blur function, thereby realizing the motion blur function.

[0259] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0260] The embodiments of the present application also provide a computer program product. When the computer program product is executed on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0261] The embodiment of the present application also provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0262] The term "user interface (UI)" in the specification and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface. After being parsed and rendered, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by <body> tags or nodes in the web page source code. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0263] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0264] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0265] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0266] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method for image motion estimation, characterized in that: Applied to an electronic device, the electronic device having a first application installed therein, the method comprising: Obtaining a rendering instruction issued by the first application, where the rendering instruction is used to instruct a graphics processor (GPU) to render image frames of the first application, where the image frames of the first application include a first image frame and a second image frame; Identifying, from the rendering instruction, a pixel position of a first object in the first image frame and a pixel position of a second object in the first image frame; wherein the first object and the second object are of different types; Identify, from the rendering instruction, a pixel position of the first object in the second image frame and a pixel position of the second object in the second image frame; Determine, by optical flow motion estimation method, a motion vector of a pixel position where the first object is located between the first image frame and the second image frame; Native motion vector information of the first application is obtained, and a motion vector of a pixel position where the second object is located between the first image frame and the second image frame is determined from the native motion vector information.

2. The method according to claim 1, characterized in that The electronic device includes a motion estimation module; The obtaining of the rendering instruction issued by the first application specifically includes: The motion estimation module obtains the rendering instruction issued by the first application from the graphics interface; The acquiring native motion vector information of the first application, and determining the motion vector of the pixel position where the second object is located between the first image frame and the second image frame from the native motion vector information specifically includes: The motion estimation module obtains native motion vector information of the first application from the graphic interface, and determines a motion vector of a pixel position where the second object is located between the first image frame and the second image frame from the native motion vector information.

3. The method according to claim 2, characterized in that The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines optical flow motion vector information between the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; The motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

4. The method according to claim 2, characterized in that: The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines an optical flow motion vector of a pixel position of the first object between the first image frame and the second image frame based on a pixel position of the first object in the first image frame, a pixel position of the second object in the second image frame, the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame to the motion estimation module; The motion estimation module uses the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame as the motion vector of the pixel position where the first object is located between the first image frame and the second image frame.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: The motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; The motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; The optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

6. The method according to any one of claims 1 to 5, characterized in that The native motion vector information is motion field motion vector information between the first image frame and the second image frame determined by the first application through a 3D motion field estimation method.

7. The method according to any one of claims 1 to 6, characterized in that The electronic device further includes a frame insertion module; and the method further includes: The motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; The interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

8. The method according to any one of claims 1 to 7, characterized in that Before identifying the pixel position of the first object in the first image frame and the pixel position of the second object in the first image frame from the rendering instruction, the method further includes: The motion estimation module determines that the native motion vector information can be obtained from the graphics interface.

9. The method according to any one of claims 1 to 7, characterized in that: The first object is a semi-transparent object, and the second object is an opaque object.

10. A method for image motion estimation, characterized in that: Applied to an electronic device, the electronic device is installed with a first application; the method comprises: Obtaining a rendering instruction issued by the first application, where the rendering instruction is used to instruct the GPU to render image frames of the first application, where the image frames of the first application include a first image frame and a second image frame; Identifying, from the rendering instruction, a pixel position of a first object in the first image frame and a pixel position of a second object in the first image frame; wherein the first object and the second object are of different types; Identify, from the rendering instruction, a pixel position of the first object in the second image frame and a pixel position of the second object in the second image frame; Determine, by optical flow motion estimation method, a motion vector of a pixel position where the first object is located between the first image frame and the second image frame; Acquire a first transformation matrix and first depth information of the first image frame, and a second transformation matrix and second depth information of the second image frame; Based on the first transformation matrix, the first depth information, the second transformation matrix and the second depth information, a motion vector of a pixel position where the second object is located between the first image frame and the second image frame is determined by a 3D motion field motion estimation method.

11. The method according to claim 10, characterized in that The electronic device includes a motion estimation module; The obtaining of the rendering instruction issued by the first application specifically includes: The motion estimation module obtains the rendering instruction issued by the first application from the graphics interface; The acquiring a first transformation matrix and first depth information of the first image frame, and a second transformation matrix and second depth information of the second image frame specifically includes: The motion estimation module obtains a first transformation matrix and first depth information of the first image frame, and a second transformation matrix and second depth information of the second image frame from the graphic interface.

12. The method according to claim 11, characterized in that The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines optical flow motion vector information between the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector information between the first image frame and the second image frame to the motion estimation module; The motion estimation module determines the motion vector of the pixel position of the first object between the first image frame and the second image frame from the optical flow motion vector information between the first image frame and the second image frame based on the pixel position of the first object in the first image frame and the pixel position of the first object in the second image frame.

13. The method according to claim 11, characterized in that The electronic device also includes an optical flow algorithm interface; Determining the motion vector of the pixel position of the first object between the first image frame and the second image frame by using the optical flow motion estimation method specifically includes: The optical flow algorithm interface determines an optical flow motion vector of a pixel position of the first object between the first image frame and the second image frame based on a pixel position of the first object in the first image frame, a pixel position of the first object in the second image frame, the first image frame and the second image frame; The optical flow algorithm interface sends the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame to the motion estimation module; The motion estimation module uses the optical flow motion vector of the pixel position where the first object is located between the first image frame and the second image frame as the motion vector of the pixel position where the first object is located between the first image frame and the second image frame.

14. The method according to claim 12 or 13, characterized in that The method further comprises: The motion estimation module obtains the buffer address of the first image frame and the buffer address of the second image frame from the graphics interface; The motion estimation module sends the buffer address of the first image frame and the buffer address of the second image frame to the optical flow algorithm interface; The optical flow algorithm interface obtains the first image frame from the buffer address of the first image frame, and obtains the second image frame from the buffer address of the second image frame.

15. The method according to any one of claims 10 to 14, characterized in that The electronic device further includes a frame insertion module; and the method further includes: The motion estimation module sends the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame to the interpolation module; The interpolation module generates a target image frame based on the first image frame, the second image frame, and the motion vector information between the first image frame and the second image frame; wherein the motion vector information includes the motion vector of the pixel position of the first object between the first image frame and the second image frame, and the motion vector of the pixel position of the second object between the first image frame and the second image frame.

16. The method according to any one of claims 10 to 15, characterized in that The first object is a dynamic object, and the second object is a static object.

17. An electronic device, characterized in that: It comprises one or more processors and one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer programs, and when the one or more processors execute the computer programs, the electronic device executes the method as described in any one of claims 1-9 or the method as described in any one of claims 10-16.

18. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes a processing circuit and an interface circuit, the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method as described in any one of claims 1-9 or the method as described in any one of claims 10-16.

19. A computer-readable storage medium, characterized in that: The method comprises a computer program, and when the computer program is executed on a processor on an electronic device, the electronic device executes the method according to any one of claims 1 to 9 or the method according to any one of claims 10 to 16.

Images