Motion vector determination method, chip, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of image processing, and provides a motion vector determination method, a chip, electronic equipment and a readable storage medium. The method comprises the steps that static pixel points and dynamic pixel points of a target image are acquired, the static pixel points are pixel points which are not moved in position in the target image, and the dynamic pixel points are pixel points which are moved in position in the target image; performing static vector acquisition operation on the static pixel points to obtain static motion vectors of the static pixel points; and performing dynamic vector acquisition operation on the dynamic pixel points to obtain dynamic motion vectors of the dynamic pixel points, the motion vectors including static motion vectors and dynamic motion vectors. The method can improve the precision of the motion vector.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a motion vector determination method, an electronic device and a readable storage medium. Background Art

[0002] In frame prediction technology, the acquisition of motion vectors is of great significance to the display effect of images. Electronic devices can determine the motion trend of pixels based on the calculated motion vectors, and obtain predicted images based on real images and motion vectors.

[0003] Frame prediction technology is often used in some applications (applications, APPs) with high refresh rates, such as some game APPs. In order to enhance the screen experience, game APPs have a higher screen refresh rate. In the development and update process of mobile game APPs, the screen refresh chip has developed from 60Hz to 90Hz, and now to 120Hz. Correspondingly, the graphics rendering resolution has also developed from 720p to 1080p, and even reached 2k. This will cause the GPU to have a very large amount of computation, and the performance requirements of the GPU are very high, and the power consumption is also very large. In order to reduce the amount of GPU computation and save power consumption, frame prediction technology is often used. For example, the middle frame image can be predicted based on the two real frames before and after, and the predicted image predicted by the middle frame can be interpolated, thereby ensuring that the game application has a high refresh rate while reducing the amount of GPU computation, thereby reducing power consumption. Then in the frame prediction technology, the accuracy of the motion vector is extremely important for the clarity and accuracy of the predicted image.

[0004] A common method of determining a motion vector is to generate a motion vector based on the position change of each pixel in a previous frame image and a next frame image. However, this method has low accuracy. Summary of the invention

[0005] The present application provides a motion vector determination method, device, chip, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of motion vectors.

[0006] In a first aspect, a motion vector determination method is provided, comprising: acquiring static pixel points and dynamic pixel points of a target image, wherein the static pixel points are pixel points whose positions in the target image have not moved, and the dynamic pixel points are pixel points whose positions in the target image have moved; performing a static vector acquisition operation on the static pixel points to obtain a static motion vector of the static pixel points; performing a dynamic vector acquisition operation on the dynamic pixel points to obtain a dynamic motion vector of the dynamic pixel points, wherein the motion vector comprises a static motion vector and a dynamic motion vector.

[0007] The GPU separates the dynamic pixels and static pixels of the real image, and then uses the respective adaptive methods for the dynamic pixels and static pixels to obtain the motion vector. Compared with the motion vector acquisition method that uses all static pixels, the motion vector calculation is more accurate; compared with the motion vector acquisition method that uses all dynamic pixels, the amount of calculation is reduced and power consumption is saved. The method provided in the embodiment of the present application separates the dynamic pixels and static pixels, and uses the respective adaptive methods to obtain the motion vector, which is more reasonable than not distinguishing between dynamic pixels and static pixels, but generally using the same method.

[0008] In a possible implementation, static pixels and dynamic pixels of a target image are obtained, including: performing a static and dynamic pixel separation operation on each pixel in the target image to obtain a static and dynamic mask image, where the static and dynamic mask image is a mask image that marks dynamic pixels and static pixels.

[0009] After the GPU obtains the previous frame of the image through the texture ID, it can first create an R8 single-channel image to mark dynamic pixels and static pixels, thereby obtaining a dynamic and static mask image (Mask image). Specifically, the GPU can perform dynamic and static pixel separation for each pixel in the previous frame of the image to determine whether the pixel is a dynamic pixel or a static pixel, and then write the determination result into the created R8 single-channel image to obtain a dynamic and static mask image. The method of marking dynamic and static pixels through a mask image is convenient to operate.

[0010] In a possible implementation, the target image is a first frame image or a second frame image, and the dynamic and static pixel separation operation includes: obtaining first two-dimensional image coordinates of a first pixel, where the first pixel is a pixel in the first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel in the first frame image; performing a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinates to obtain first three-dimensional world coordinates of the first pixel; projecting the first three-dimensional world coordinates into the second frame image to obtain three-dimensional projection coordinates; obtaining two-dimensional projection coordinates according to the three-dimensional projection coordinates; performing a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the two-dimensional projection coordinates to obtain second three-dimensional world coordinates; if the distance between the first three-dimensional world coordinates and the second three-dimensional world coordinates is greater than or equal to a preset distance threshold, the first pixel is a dynamic pixel; if the distance between the first three-dimensional world coordinates and the second three-dimensional world coordinates is less than the preset distance threshold, the first pixel is a static pixel.

[0011] The GPU converts the 2D image coordinates of the first pixel into a 3D world coordinate system and combines it with depth information. It then uses a world matrix containing rotation parameters and translation parameters to calculate the 3D projection coordinates of the pixel, and the obtained coordinates are highly accurate. The 2D projection coordinates are extracted based on the high-precision 3D projection coordinates. When the 2D projection coordinates are converted into the 3D world coordinates corresponding to the first frame of the image, the depth information of the other frame of the image is combined, and the accuracy of the obtained 3D world coordinates is also high. Based on this, the accuracy of the separated static and dynamic pixels is high, which in turn makes the accuracy of the motion vector also high.

[0012] In a possible implementation, a static vector acquisition operation includes: acquiring first two-dimensional image coordinates of a first pixel point, where the first pixel point is a pixel point in a first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel point in the first frame image; performing a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinates to obtain first three-dimensional world coordinates of the first pixel point; projecting the first three-dimensional world coordinates into a second frame image to obtain three-dimensional projection coordinates; and obtaining a static motion vector based on the first two-dimensional image coordinates and the three-dimensional projection coordinates.

[0013] In the above-mentioned method of acquiring the static motion vector of the static pixel point, by projecting the two-dimensional image coordinates into three-dimensional world coordinates and combining the depth value of the depth image, the coordinate position has a higher accuracy during the coordinate projection and transformation process. Afterwards, the GPU obtains a high-accuracy static motion vector based on the first two-dimensional image coordinates and the obtained precise three-dimensional projection coordinates. At the same time, since the static motion vector is directly determined based on the vector difference between the three-dimensional projection coordinates and the two-dimensional image coordinates, the amount of calculation is reduced compared to the method of combining the camera's projection matrix and the world matrix to determine the motion vector, thereby saving power consumption.

[0014] In a possible implementation, obtaining a static motion vector according to the first two-dimensional image coordinates and the three-dimensional projection coordinates includes: extracting two-dimensional projection coordinates from the three-dimensional projection coordinates; and obtaining the static motion vector according to the difference between the two-dimensional projection coordinates and the first two-dimensional image coordinates.

[0015] In a possible implementation, the two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation includes: obtaining the target three-dimensional world coordinates according to the target two-dimensional image coordinates, the depth value of the target pixel point in the target image, the projection matrix and the world matrix of the target image, wherein the target pixel point is the pixel point in the target image, and the target two-dimensional image coordinates are the two-dimensional image coordinates of the target pixel point in the target image; when the target image is a first frame image, the target three-dimensional world coordinates are the first three-dimensional world coordinates; when the target image is a second frame image, the target three-dimensional world coordinates are the second three-dimensional world coordinates.

[0016] In this method, the 2D image coordinates of the first pixel are converted into a 3D world coordinate system and combined with the depth information, and then the 3D projection coordinates of the pixel are calculated using a world matrix containing rotation parameters and translation parameters, so that the obtained coordinates have high accuracy. The 2D projection coordinates are extracted based on the high-precision 3D projection coordinates, and when the 2D projection coordinates are converted into the 3D world coordinates corresponding to the first frame of the image, the depth information of the other frame of the image is combined, so that the accuracy of the obtained 3D world coordinates is also high. Finally, the accuracy of the static motion vector determined based on the high-precision 3D world coordinates corresponding to the previous and next frames is also high.

[0017] In a possible implementation, projecting the first three-dimensional world coordinates into the second frame image to obtain the three-dimensional projection coordinates includes: projecting the first three-dimensional world coordinates into the second frame image according to the projection matrix and the world matrix of the second frame image to obtain the three-dimensional projection coordinates.

[0018] In a possible implementation, the target image is a first frame image or a second frame image, and the dynamic vector acquisition operation includes: acquiring a first brightness value of a first pixel point, where the first pixel point is any pixel point in the first frame image; converting the first brightness value into a first feature vector; screening out a first matching pixel point that matches the first pixel point from a second pixel point set, where the second pixel point set is a set of pixels in the second frame image, a vector distance between the first matching feature vector and the first feature vector is smaller than a vector distance between other feature vectors and the first feature vector, the first matching feature vector is a feature vector converted from the brightness value of the first matching pixel point, other pixels are pixels in the second frame image other than the first matching pixel point, and other feature vectors are feature vectors converted from the brightness values ​​of other pixels; determining the dynamic motion vector corresponding to the first pixel point according to the difference between the two-dimensional image coordinates of the first matching pixel point in the second frame image and the two-dimensional image coordinates of the first pixel point in the first frame image.

[0019] Using the above method, the GPU generates brightness values ​​for each pixel, converts the brightness values ​​into feature vectors, and then determines matching pixel pairs based on the feature vectors that characterize the brightness characteristics of the pixels. This can accurately characterize the characteristics of each pixel, so that the matching degree of the matched pixel pairs is high, thereby making the accuracy of the dynamic motion vector high.

[0020] In a possible implementation, the target image is a first frame image or a second frame image, and the dynamic vector acquisition operation includes: acquiring a first image pyramid, the first image pyramid is an image pyramid generated according to the first frame image, the first image pyramid includes an N-order first brightness image, wherein the i-th order first brightness image is a brightness image obtained by downsampling the i+1-th order first brightness image, i is a positive integer greater than 0 and less than N, the brightness value of a first downsampled pixel point in the i-th order first brightness image is an average of the brightness values ​​of four pixels corresponding to the position of the first downsampled pixel point in the i+1-th order first brightness image, and the first downsampled pixel point is any pixel in the i-th order first brightness image. a pixel point; obtaining a second image pyramid, the second image pyramid is an image pyramid generated according to the second frame image, the second image pyramid includes an N-order second brightness image, wherein the i-th order second brightness image is a brightness image obtained by downsampling the i+1-th order second brightness image, i is a positive integer greater than 0 and less than N, and the brightness value of the second downsampled pixel point in the i-th order second brightness image and the average brightness value of four pixels corresponding to the position of the second downsampled pixel point in the i+1-th order second brightness image, the second downsampled pixel point is any pixel point in the i-th order second brightness image; determining the j-th order first brightness image from the first image pyramid, and obtaining the j-th order first brightness image from the second image pyramid ... The j-th order second brightness image is determined in the pyramid, wherein j is a positive integer greater than 0 and less than N; according to the first pixel point and the first corresponding pixel point, an initial motion vector corresponding to the first pixel point is determined, the first pixel point is any pixel point in the j-th order first brightness image, the vector distance between the first brightness vector and the first corresponding brightness vector is less than the vector distance between the first brightness vector and other brightness vectors, the first brightness vector is a vector obtained by converting the brightness value of the first pixel point, the first corresponding brightness vector is a vector obtained by converting the brightness value of the first corresponding pixel point, other brightness vectors are vectors obtained by converting the brightness values ​​of other pixel points, and other pixel points are pixels in the j-th order second brightness image that are not the first corresponding pixel points. pixel point; according to the two-dimensional image coordinates of the first pixel point in the first brightness image of the jth order, and the two-dimensional image coordinates of the first corresponding pixel point in the second brightness image of the jth order, obtain the initial dynamic motion vector; search for the first intermediate pixel point with the smallest brightness value difference with the first pixel point in the first brightness image of the j+1th order; search for the first corresponding intermediate pixel point with the smallest brightness value difference with the first corresponding pixel point in the second brightness image of the j+1th order; according to the two-dimensional image coordinates of the first intermediate pixel point in the first brightness image of the j+1th order, and the two-dimensional image coordinates of the first corresponding intermediate pixel point in the second brightness image of the j+1th order, correct the initial motion vector to obtain the intermediate dynamic motion vector;If j is less than N-1, j+1 is used as the new j, and the step of searching for the first intermediate pixel with the smallest brightness difference from the first pixel in the first brightness image of the j+1th order is continued until j+1=N, and the dynamic motion vector is obtained. ;

[0021] The above i and j can be the same or different. Wherein, when the i-th order represents an arbitrary first-order brightness image, the i+1-th order represents a brightness image one order higher than the i-th order in the same image pyramid. When the j-th order represents an arbitrary first-order brightness image, the j+1-th order represents a brightness image one order higher than the j-th order in the same image pyramid. Wherein, for the first image pyramid, the first down-sampled pixel point is a pixel point in the lower-order brightness image, and the brightness value of the first down-sampled pixel point is the average brightness value of the four pixel points at the corresponding positions in the higher-order brightness image. For the second image pyramid, the second down-sampled pixel point is a pixel point in the lower-order brightness image, and the brightness value of the second down-sampled pixel point is the average brightness value of the four pixel points at the corresponding positions in the higher-order brightness image in the second image pyramid.

[0022] The GPU can specify a lower-order brightness image to start the calculation, for example, a brightness image with an order lower than half of N. The smaller the selected j, the less the amount of calculation; the larger the selected j, the higher the accuracy. In actual use, you can take both accuracy and amount of calculation into consideration to select a reasonable value.

[0023] In this method, the GPU generates image pyramids for the first and second frames based on the brightness values ​​of the pixels, and then selects an intermediate image of the same order and low order (i.e., an image with fewer pixels and smaller image size) from each of the two image pyramids to calculate the initial dynamic motion vector. Figure 8In the embodiment shown, the matching pixel pair is obtained by searching the first matching pixel for the first pixel. Next, the GPU can find the local optimal solution in the higher-order image of the intermediate image where each pixel is located based on the pixel in the pixel pair, that is, find a pixel with the closest brightness value. Then, the GPU obtains the offset between the coordinates of the pixel with the closest brightness value and the original pixel on the intermediate image, and uses this offset to correct the initial dynamic motion vector obtained previously. It should be noted that such a correction method can be iterated step by step until the highest-order image. The local optimal solution in this embodiment is the pixel with the closest brightness value in the local area. In this method, the initial dynamic motion vector can be obtained by calculating a small number of feature vectors in the low-order image, and then the dynamic motion vector is corrected step by step according to the offset of the pixel of each order image until the highest-order image, thereby ensuring the accuracy of the dynamic motion vector while avoiding the budget of a large number of feature vectors. Instead, the accurate dynamic motion vector can be obtained by finding the local optimal solution of simple calculation, which reduces the amount of calculation and saves power consumption.

[0024] In a possible implementation, j is a positive integer less than half of N.

[0025] When the GPU uses a positive integer j that is less than half of N, it can ensure that the amount of calculation is not too large, thereby effectively saving power consumption.

[0026] In a second aspect, a motion vector determination device is provided, comprising a unit composed of software and / or hardware, wherein the unit is used to execute any one of the methods in the technical solution of the first aspect.

[0027] In a third aspect, an embodiment of the present application provides a chip, comprising a processor; the processor is used to read and execute a computer program stored in a memory to execute any one of the methods in the technical solution described in the first aspect.

[0028] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0029] Optionally, the chip is a graphics processing unit GPU.

[0030] Further optionally, the chip also includes a communication interface.

[0031] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor, a memory and an interface; the processor, the memory and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution described in the first aspect.

[0032] In a fifth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising any chip in the technical solution as described in the third aspect.

[0033] In a sixth aspect, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution described in the first aspect.

[0034] In a seventh aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is run on an electronic device, the electronic device executes any one of the methods in the technical solution described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of the present application;

[0036] Figure 2 is a software structure block diagram of the terminal device 100 provided in an embodiment of the present application;

[0037] Figure 3 This is an example of a game video frame rendering and display process provided by an embodiment of the present application;

[0038] Figure 4 is a schematic diagram of various coordinate systems provided in an embodiment of the present application;

[0039] Figure 5 is a flowchart of a method for determining a motion vector provided in an embodiment of the present application;

[0040] Figure 6 This is a flow chart of a method for separating moving and static pixels provided in an embodiment of the present application;

[0041] Figure 7 is a flowchart of a static motion vector acquisition operation method provided in an embodiment of the present application;

[0042] Figure 8 is a flowchart of a method for obtaining a dynamic motion vector provided in an embodiment of the present application;

[0043] Fig. 9 is a schematic diagram of an image pyramid provided in an embodiment of the present application;

[0044] Fig.10 This is a schematic diagram of the distribution position of pixels corresponding to an example provided in an embodiment of the present application;

[0045] Fig.11is a schematic diagram of the relative positions of corresponding pixel points in an example pixel block provided in an embodiment of the present application;

[0046] Fig.12 is a flowchart of another method for determining a motion vector provided in an embodiment of the present application;

[0047] Fig.13 It is a structural diagram of a motion vector determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0049] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0050] The motion vector determination method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0051] For example, Figure 11 is a schematic diagram of the structure of an example of a terminal device 100 provided in an embodiment of the present application. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0052] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0053] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0054] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0055] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0056] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0057] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0058] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0059] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0060] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0061] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0062] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0063] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0064] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0065] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.

[0066] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0067] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0068] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 1The structures of antenna 1 and antenna 2 in the figure are only an example. Each antenna in the terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0069] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0070] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be arranged in the same device as the mobile communication module 150 or other functional modules.

[0071] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0072] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a BeiDou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0073] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0074] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0075] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0076] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0077] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0078] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0079] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0080] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0081] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0082] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0083] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0084] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0085] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0086] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0087] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0088] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0089] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0090] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0091] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0092] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0093] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.

[0094] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0095] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0096] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0097] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0098] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0099] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0100] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0101] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0102] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0103] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0104] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0105] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0106] Figure 2 1 is a software structure diagram of the terminal device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0107] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0108] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0109] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0110] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0111] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0112] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0113] The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (including connection, disconnection, etc.).

[0114] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0115] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0116] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0117] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0118] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0119] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0120] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0121] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0122] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0123] A 2D graphics engine is a drawing engine for 2D drawings.

[0124] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0125] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 1 and Figure 2 Taking the terminal device with the shown structure as an example, the motion vector determination method provided in the embodiment of the present application is specifically explained in combination with the accompanying drawings and application scenarios.

[0126] Motion vectors are usually used in frame prediction technology. Electronic devices can determine the motion trend of pixels based on the calculated motion vectors, and implement the process of obtaining predicted images based on real images and motion vectors. Frame prediction technology is often used in some applications (applications, APPs) with high refresh rates, such as some game APPs. In order to enhance the picture experience, game APPs have a higher screen refresh rate. In the development and update process of mobile game APPs, the screen refresh chip has developed from 60Hz to 90Hz, and now to 120Hz. Correspondingly, the graphics rendering resolution has also developed from 720p to 1080p, and even reached 2k. This will cause the GPU to have a very large amount of computation, and the performance requirements of the GPU are very high, and the power consumption is also very large. In order to reduce the GPU's computational workload and save power consumption, frame prediction technology is often used. For example, the middle frame image can be predicted based on the previous and next two frames of real images, and the predicted image predicted by the middle frame can be interpolated, thereby ensuring that the game application has a high refresh rate while reducing the GPU's computational workload, thereby reducing power consumption.

[0127] In frame prediction technology, how to accurately determine the position of pixels in the predicted image is an important factor in ensuring that the predicted image is displayed clearly and accurately. A common method is to generate a motion vector based on the position change of each pixel in the previous frame image and the next frame image, and then move and render the pixels in the real image based on the motion vector to generate a predicted image, but this method is computationally intensive.

[0128] In the technical solution of the present application, by obtaining the feature points of the previous frame image and the feature points of the next frame image, and then extracting the motion vector based on the matched feature points of the previous and next frames, the number of pixels to be processed can be greatly reduced, thereby reducing the amount of calculation, improving the calculation efficiency, and saving power consumption. The obtained motion vector is then applied to the global image, and the motion information of the full-screen pixels can be estimated with less calculation, thereby generating a predicted image.

[0129] The technical solution of the embodiment of the present application can be used in the AR / VR field, and can also be used in the game application field. Next, the application in the frame prediction of the game video played by the game application is used as an example for explanation. Figure 3 The rendering and display process of the game video frame provided by the embodiment of the present application is shown.

[0130] The game application runs on the operating system of the terminal device. In the game scene, the game application modifies the function pointer of the rendering API to generate the game's rendering instruction stream, such as hooking the game's rendering instruction stream. The game's rendering instruction stream is passed to the GPU through the graphics manager and GPU driver in turn. The GPU obtains the function through the rendering instruction and identifies the game's rendering instruction stream based on the characteristics of the game's rendering instruction.

[0131] The rendering instruction stream is an instruction stream composed of multiple drawing instructions (rendering instruction sequence). Generally, a rendering instruction stream is used to draw a frame of image. Specifically, the GPU can execute each drawing instruction in the rendering instruction stream to realize the drawing of each object in a frame of image. The set of objects generated according to the rendering instruction stream can form a frame of image. A frame of image is a frame of static image displayed by the game application. Each frame of static image can be composed of 3D scene image and UI image, etc., wherein the 3D scene image is also called color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects and skills, etc., and the UI image may include control buttons (such as running buttons and skill buttons, etc.), small maps, floating text, user avatars, energy pack controls and blood bars, etc., and the UI image is located on the upper layer of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image belong to the objects in the game frame. In addition, through the rendering instruction stream, the world matrix and projection matrix of the virtual camera corresponding to a frame of image, as well as general information such as the depth image corresponding to a frame of image can also be obtained.

[0132] The game frames in the embodiment of the present application can be divided into two types:

[0133] One is the real frame generated according to the rendering instruction stream, which is the real image mentioned above. In the game scene, the operating system sends rendering instruction streams to the GPU in sequence. The GPU generates a real frame corresponding to each rendering instruction stream in sequence according to the order in which these rendering instruction streams are received. Among them, one rendering instruction stream is used to draw one real frame.

[0134] The other is a predicted frame generated based on two rendering instruction streams obtained successively, which is the predicted image mentioned above. "Two rendering instruction streams obtained successively" means obtaining rendering instruction stream 1 first, and then obtaining rendering instruction stream 2. Specifically, the GPU can obtain two rendering instruction streams successively (one rendering instruction stream is used to draw a real frame), and separate the 3D scene image and UI image of two adjacent frames from each rendering instruction stream, as well as obtain the depth image corresponding to each real frame, the world matrix and projection matrix of the game virtual camera and other general information, and then use this information to calculate the motion vector of each pixel, and then perform pixel-by-pixel offset on the 3D scene image corresponding to a real frame based on the motion vector. Afterwards, the GPU can call the neural network model of the NPU to perform image repair on the blank area after the pixel-by-pixel offset, and superimpose the repaired 3D scene image with the UI image corresponding to this real frame to generate the final predicted frame.

[0135] Assuming that the GPU generates real frame N-1 according to the rendering instruction stream 1 obtained first, generates real frame N+1 according to the rendering instruction stream 2 obtained later, and generates predicted frame N according to rendering instruction streams 1 and 2, the GPU can send these frames to the queue to be displayed in the order of real frame N-1, predicted frame N and real frame N+1. Then the display screen plays real frame N-1, predicted frame N and real frame N+1 in sequence.

[0136] It should be understood that since the resource consumption of generating predicted frames using inter-frame prediction technology is less than the resource consumption of rendering real frames by game applications, by reducing the number of real frames generated by game applications and using inter-frame prediction technology to generate predicted frames, not only the total amount of game frames can be guaranteed, but also the power consumption load of the chip can be reduced.

[0137] For example, in a game scenario, the GPU can continuously obtain a rendering instruction stream from the game application. Whenever a rendering instruction stream is obtained, the GPU can perform the following steps 1 and 2 for the rendering instruction stream.

[0138] Step 1: The GPU separates the 3D scene image and the UI image from the rendering instruction stream.

[0139] The 3D scene image mentioned above refers to a color image that does not contain UI pixels but contains a 3D scene, that is, an RGB image.

[0140] For example, the above-mentioned rendering instruction stream may be a Hook game rendering instruction stream. The GPU may identify the various instructions contained in the Hook game rendering instruction stream through the rendering instruction acquisition function according to the characteristics of the game's rendering instruction stream. The Hook game rendering instruction stream generally includes a variety of drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Among them, the scene drawing instructions can be used to draw 3D scene images such as scenery, characters, special effects, and skills in the game, the UI drawing instructions can be used to draw UI images such as control buttons, small maps, and floating text, and the image display instructions are used to put the drawn image data into the specified location of the operating system (such as framebuffer0) to complete the frame display.

[0141] The pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located on the upper layer of the 3D scene image. If the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, the object in the UI image may cause visual occlusion to the object in the 3D scene image. The object in the 3D scene image may change its pixel coordinates for two reasons: one reason is that the position of objects such as characters in the game scene moves; the other reason is that the virtual camera in the game scene shakes, that is, the character's perspective changes. When the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, regardless of the above reasons for changing the pixel coordinates, if the object in the 3D scene image is moved according to the following steps 3-5, the object in the UI image will move with the object in the 3D scene image, but in fact the object in the UI image may not move, or the motion vectors of the object in the UI image and the object in the 3D scene image are different, which will cause the final predicted frame to have a more serious image quality problem, so the GPU needs to separate the 3D scene image and the UI image from the rendering instruction stream.

[0142] Specifically, the GPU can identify whether it is a UI drawing instruction based on the characteristics of the game rendering instruction. For example, the UI drawing instruction may include resource binding instructions, state setting instructions, and drawing instructions, and point the output of these instructions to a newly created image. The newly created image can be used to save UI pixels, so the newly created image can be called a UI image. In addition, the GPU can change the original rendering generation result to a newly created image for saving a 3D scene image that does not contain UI pixels. In this way, the GPU separates the UI image and the 3D scene image from the rendering instruction stream.

[0143] Step 2: The GPU obtains the depth image, the world matrix and the projection matrix of the virtual camera according to the rendering instruction stream.

[0144] The GPU can identify the rendering instruction sequence for model rendering based on the characteristics of the game rendering instructions, and determine the glBindBuffer instruction located before the model rendering drawcall instruction. Among them, the glBindBuffer instruction contains the relevant data of the virtual camera. The GPU can obtain the world matrix and projection matrix of the virtual camera from the buffer corresponding to the glBindBuffer instruction. In addition, the GPU can also obtain the depth image from the rendering instruction stream.

[0145] It should be noted that the 3D scene image obtained by the GPU from step 1 and the depth image obtained from step 2 have the same image content, for example, both images contain people, trees, and mountains. The difference between the two images is that the 3D scene image is a three-channel (RGB) color image; the depth image is a single-channel image, which is similar to a grayscale image. Each pixel value of the depth image can be used to represent the distance between the virtual camera and the object, and the depth image reflects the geometric shape of the visible surface of the object.

[0146] In order to facilitate the understanding of the world matrix and projection matrix of the above virtual camera, the following Figure 4 The pixel coordinate system, image coordinate system, normalized device coordinate system (NDC), camera coordinate system and world coordinate system involved in the embodiments of the present application are explained.

[0147] The pixel coordinate system o-wh and the image coordinate system o-xy are two-dimensional coordinate systems. Usually, images are stored in the form of arrays in terminal devices, and the value of each pixel in the array represents the brightness of the image point. The origin of the pixel coordinate system o-wh is located in the upper left corner of the image, the w axis is horizontal to the right, and the h axis is vertically downward. Each pixel coordinate (w,h) is the number of columns and rows of a pixel in the array, so (w,h) is the coordinate of the pixel coordinate system o-wh in pixels. The origin of the image coordinate system o-xy is located at the center of the image (called the principal point of the image), the x axis is parallel to the w axis, and the y axis is parallel to the h axis. The coordinates in the pixel coordinate system o-wh and the image coordinate system o-xy can be converted to each other.

[0148] The standardized device coordinate system (NDC coordinate system) o-x'y'z' is a three-dimensional coordinate system. In the standardized device coordinate system o-x'y'z', the values ​​of x', y' and z' are in the interval [-1,1], where z' represents the depth value of the image, and x' and y' are obtained by normalizing the coordinates (x, y) in the image coordinate system o-xy, respectively. Through the viewport transform, the standardized device coordinate system o-x'y'z' can be transformed into the image coordinate system o-xy or the pixel coordinate system o-wh. Taking a frame of 512*512 size image as an example, the pixel coordinates of the lower right corner point in the pixel coordinate system o-wh are (512, 512). After the coordinate transformation, the coordinates of the point in the standardized device coordinate system o-x'y'z' are (1, 1, z'), where z' represents the depth value of the coordinate. Optionally, z' can also be a normalized depth value. Converting the image coordinate system into a standardized coordinate system is equivalent to normalization.

[0149] The camera coordinate system O0-X0Y0Z0 is a three-dimensional coordinate system. In the camera coordinate system O0-X0Y0Z0, the virtual camera is a virtual camera set for the game scene, not a real physical camera. In the game scene, if there are multiple players roaming in the game world, since each player has a different position and sight direction in the game world, the frame displayed by the terminal device used by each player is also different. Suppose a coordinate system is created with a certain player as the origin. This coordinate system can be called the camera coordinate system, the observation space coordinate system or the three-dimensional camera coordinate system. Figure 4 , the origin O0 of the camera coordinate system is the optical center (projection center) of the virtual camera, the X0 axis and the Y0 axis are parallel to the x-axis and y-axis of the camera plane, the Z0 axis is the optical axis of the virtual camera, the Z0 axis is perpendicular to the camera plane, and oO0 is the focal length of the virtual camera. It should be understood that each player has a different position and sight direction in the game world, so the camera coordinate system O0-X0Y0Z0 established with the virtual camera corresponding to each player as the center will also be different.

[0150] The world coordinate system O-XYZ is a three-dimensional coordinate system introduced to describe the position of each virtual camera. In the game scene, virtual objects such as characters, tables, rooms, animals and vehicles have a specific position in the game. The position of the virtual object in the game world is defined as the coordinate in the world coordinate system, which is called the world space coordinate (also called the three-dimensional world coordinate). For example, the world coordinate system O-XYZ can be a three-dimensional coordinate system established with the center of the earth as the origin. It should be understood that different players have different positions and sight directions in the world coordinate system O-XYZ, which makes the position and orientation of the virtual camera in the world coordinate system O-XYZ also different. The coordinates in the camera coordinate system O0-X0Y0Z0 can be converted into coordinates in the world coordinate system O-XYZ through translation and rotation.

[0151] The world matrix (viewMatrix), also known as the external parameter matrix, is used to convert the object coordinates in the world coordinate system O-XYZ into the camera coordinate system O0-X0Y0Z0. That is, when the perspective of the virtual camera is used as the origin of the game scene, the world matrix transforms all world space coordinates into observation coordinates relative to the position and direction of the virtual camera.

[0152] The projection matrix (projectionMatrix), also known as the intrinsic parameter matrix, is used to convert the object coordinates in the camera coordinate system O0-X0Y0Z0 into descriptions using the standardized device coordinate system o-x'y'z'. When rendering, a three-dimensional scene is usually projected onto the screen (i.e., the camera surface) to become a two-dimensional image. Specifically, all vertex data is first converted from the camera coordinate system O0-X0Y0Z0 to the clipping space, and then the converted clipping coordinates are transformed into the standardized device coordinate system o-x'y'z' by dividing by the w component in the clipping coordinate system. Among them, the camera coordinate system O0-X0Y0Z0 is a right-handed coordinate system, and the standardized device coordinate system o-x'y'z' is a left-handed coordinate system. The vertex data is converted from the camera coordinate system to the clipping space coordinate system to achieve linear transformation and projection transformation, and then divided by the w component in the clipping coordinate system to achieve the conversion from homogeneous coordinates to non-homogeneous coordinates.

[0153] Assume that the GPU obtains rendering instruction stream 1 and rendering instruction stream 2 in sequence. Among them, rendering instruction stream 1 can be used to generate real frame N-1, and rendering instruction stream 2 can be used to generate real frame N+1. According to the above steps 1 and 2, the GPU can obtain the 3D scene image Color0, UI image UI0, depth image Depth0, virtual camera's world matrix ViewMatrix0 and virtual camera's projection matrix ProjectionMatrix0 corresponding to the real frame N-1, and the 3D scene image Color1, UI image UI1, depth image Depth1, virtual camera's world matrix ViewMatrix1 and virtual camera's projection matrix ProjectionMatrix1 corresponding to the real frame N+1. Then, the GPU can obtain the motion vector according to the following method.

[0154] Next, how the present application determines the motion vector is described in detail with reference to the accompanying drawings. It should be noted that the motion vector can be used for various business needs such as image editing, image drawing, and image restoration, and the present application embodiment does not limit this. The executor of the technical solution of the present application embodiment can be a terminal device, a GPU in the terminal device, or other software or hardware functional modules in the terminal device.

[0155] Specifically, Figure 5 A flowchart of a method for determining a motion vector provided in an embodiment of the present application may specifically include:

[0156] S501, acquiring static pixel points and dynamic pixel points of a target image, wherein the static pixel points are pixel points whose positions in the target image have not moved, and the dynamic pixel points are pixel points whose positions in the target image have moved.

[0157] Usually, motion vectors can be calculated based on two consecutive frames, such as the previous frame and the next frame. The previous frame can be considered as the N-1 frame, and the next frame can be considered as the N+1 frame, both of which are real images, that is, images calculated by the game application. Specifically, after obtaining the previous frame and the next frame, the GPU can use the previous frame as the target image, or the next frame as the target image. This embodiment of the present application does not limit this, and the target image is described here as the previous frame.

[0158] Optionally, the GPU can determine whether each pixel in the previous frame is a dynamic pixel or a static pixel. A dynamic pixel refers to a pixel whose position has changed in the next frame relative to the previous frame, such as pixels related to objects such as running people and aircraft; a static pixel refers to a pixel whose position has not changed in the next frame relative to the previous frame, such as pixels related to objects such as trees and houses in a game scene.

[0159] It should be noted that, the position has not changed in the embodiments of the present application, which means that the position change is less than a certain threshold value, for example, the coordinate offset is less than a small value such as 0.5, 0.3, etc., and the position change can be ignored, then it can be said that the position has not changed, and the pixel point is a pixel point of a static object; correspondingly, the position has changed in the embodiments of the present application, which means that the position change is greater than or equal to a certain threshold value, for example, the coordinate offset is greater than or equal to 0.5, 1, etc., and the impact of the position change cannot be ignored, then it can be said that the position has changed, and the pixel point is a pixel point of a moving object.

[0160] S502: Perform a static vector acquisition operation on the static pixel point to obtain a static motion vector of the static pixel point.

[0161] Specifically, if the position of the static pixel in the image does not change and the position change relationship is simple, the GPU can be used to perform a static vector acquisition operation on the static pixel to obtain a static motion vector of the static pixel. Optionally, the static vector acquisition operation can be performed on each static pixel to obtain a static motion vector; or the static vector acquisition operation can be performed on a portion of the static pixels to obtain a static motion vector.

[0162] S503 , performing a dynamic vector acquisition operation on the dynamic pixel point to obtain a dynamic motion vector of the dynamic pixel point, where the motion vector includes a static motion vector and a dynamic motion vector.

[0163] Specifically, the position of the dynamic pixel point in the image has changed, and the position change relationship is more complicated than that of the static pixel point. Therefore, the change characteristics of the dynamic pixel point can be considered, and the GPU can be used to perform a dynamic vector acquisition operation on the dynamic pixel point to obtain the dynamic motion vector of the dynamic pixel point. Optionally, the dynamic vector acquisition operation can be performed on each dynamic pixel point to obtain the dynamic motion vector; or the dynamic vector acquisition operation can be performed on some dynamic pixels to obtain the dynamic motion vector.

[0164] It should be noted that the static motion vector of the static pixel point and the dynamic motion vector of the dynamic pixel point are combined as the total motion vector, which can be used for frame prediction technology.

[0165] Above Figure 5 In the illustrated embodiment, the GPU separates the dynamic pixels and static pixels of the real image, and then uses respective adaptive methods for the dynamic pixels and static pixels to obtain motion vectors. Compared with the motion vector acquisition method using only static pixels, the motion vector calculation is more accurate; compared with the motion vector acquisition method using only dynamic pixels, the amount of calculation is reduced and power consumption is saved. The method provided in the embodiment of the present application separates the dynamic pixels and static pixels, and uses respective adaptive methods to obtain motion vectors. Compared with not distinguishing between dynamic pixels and static pixels, but generally using the same method, it is more reasonable.

[0166] Optionally, continuing to describe with the target image being the previous frame image as an example, a possible implementation method of the above step S501 may include: performing a dynamic and static pixel separation operation on each pixel in the target image to obtain a dynamic and static mask image, where the dynamic and static mask image is a mask image that marks dynamic pixels and static pixels.

[0167] Specifically, the GPU can directly obtain the separated dynamic pixels and static pixels, and can also separate them by itself. The separation method may include: after the GPU obtains the previous frame image through the texture ID, it can first create an R8 single-channel image to mark the dynamic pixels and static pixels, thereby obtaining a dynamic and static mask image (Mask image). Specifically, the GPU can perform dynamic and static pixel separation for each pixel in the previous frame image to determine whether the pixel is a dynamic pixel or a static pixel, and then write the determined result into the created R8 single-channel image to obtain a dynamic and static mask image. In the dynamic and static mask image, if the pixel marked 1 is a dynamic pixel, then the pixel marked 0 is a static pixel; optionally, if the pixel marked 0 is a dynamic pixel, then the pixel marked 1 is a static pixel, as long as the dynamic pixel and the static pixel can be separated.

[0168] Optionally, the above-mentioned dynamic and static pixel separation operation can select any one of the pixels, and record it as the first pixel as an example for exemplary description, which can be seen in FIG. Figure 6 The method shown comprises:

[0169] S601 . Obtain first two-dimensional image coordinates of a first pixel point, where the first pixel point is a pixel point in a first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel point in the first frame image.

[0170] Here, the first frame image may be the previous frame image, and the second frame image may be the next frame image; if the first frame image is the next frame image, the second frame image may be the previous frame image. In this embodiment, the first frame image may be the previous frame image, and the second frame image may be the next frame image.

[0171] The GPU can directly obtain the two-dimensional image coordinates of a pixel point in the previous frame image, for example, obtain the two-dimensional image coordinates of the first pixel point in the previous frame image, and record them as the first two-dimensional image coordinates.

[0172] S602: Perform a two-dimensional image coordinate-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinate to obtain a first three-dimensional world coordinate of a first pixel point.

[0173] The GPU can perform a 2D image coordinate-3D world coordinate conversion operation on the first 2D image coordinate to obtain the first 3D world coordinate of the first pixel. Specifically, the GPU obtains the depth value of the first pixel from the depth image of the previous frame image, and then combines the first 2D image coordinate, the previous frame image world matrix and the projection matrix to convert the 3D world coordinate of the first pixel, which is recorded as the first 3D world coordinate.

[0174] Optionally, the GPU can obtain Depth0 of the previous frame image, obtain the depth value Depth0(u0,v0) of the first pixel in Depth0, and then multiply the two-dimensional image coordinates of the first pixel in the previous frame image by 2.0, and subtract the difference of 1.0 as the first two values ​​of the three-dimensional space coordinates expressed in the form of a four-dimensional array, and use the depth value Depth0(u0,v0) as the third value of the four-dimensional array, 1.0 as the fourth value to fill the four-dimensional array, and then multiply the three-dimensional space coordinates by the inverse matrix of the product of the world matrix viewMatrix1 of the previous frame image and the projection matrix projectionMatrix to obtain a three-dimensional coordinate matrix, and then divide the three-dimensional coordinate matrix by the w component in the clipping coordinate system to obtain the three-dimensional world coordinates worldPos1.xyz of the first pixel in the world three-dimensional coordinate system, so as to realize the coordinate transformation from two-dimensional image coordinates to three-dimensional world coordinates. Optionally, the following formula or a variation of the formula can be used:

[0175] tempData=[(u0,v0)*2.0-1.0,Depth0(u0,v0),1.0]*Inverse(viewMatrix1*projectionMatrix);

[0176] worldPos1.xyz=(tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w);

[0177] Among them, dividing by tempData.w means dividing by the w component in the clipping coordinate system; tempData represents three-dimensional coordinates, x, y and z represent the parameters of the three directions of the x, y and z three-dimensional space respectively; Inverse means performing an inverse operation on the product of the projection matrix and the world matrix, that is, the inverse matrix.

[0178] S603: Project the first three-dimensional world coordinates onto the second frame image to obtain three-dimensional projection coordinates.

[0179] The GPU can reproject the first 3D world coordinates generated in the previous step into the next frame image to obtain the 3D projection coordinates. Specifically, the GPU can project the first 3D world coordinates into the second frame image according to the projection matrix and the world matrix of the second frame image to obtain the 3D projection coordinates.

[0180] Optionally, the GPU may obtain a three-dimensional projection coordinate by multiplying the first three-dimensional world coordinate by the projection matrix and the world matrix of the next frame image.

[0181] It should be noted that the world matrix includes rotation parameters (rotation) and translation parameters (translation). Optionally, the following formula or a variation of the formula may be used:

[0182] tempData=[worldPos1.xyz,1.0]*viewMatrix2*projectionMatrix;

[0183] tempUVData.xyz=(tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w);

[0184] Among them, dividing by tempData.w represents the operation of dividing by the w component in the clipping coordinate system; tempData.xyz represents three-dimensional coordinates, x, y and z represent parameters in three directions of the three-dimensional space of x, y and z respectively, and * represents multiplication or generalized multiplication.

[0185] S604: Obtain two-dimensional projection coordinates according to the three-dimensional projection coordinates.

[0186] Specifically, the GPU can obtain the two-dimensional projection coordinates (u1, v1) from the three-dimensional projection coordinates. You can select the values ​​corresponding to the x and y directions from the above tempUVData.xyz, add 1.0 to each, and multiply by 0.5 or a value close to 0.5 to complete the linear transformation as the value of (u1, v1). It can be expressed as the following formula or formula variation:

[0187] (u1,v1)=(tempUVData.xy+1.0)*0.5;

[0188] Among them, tempUVData.xy means that the value of (u1, v1) is the value corresponding to the x and y directions in tempUVData.xyz, and the two-dimensional projection coordinates are obtained.

[0189] S605 , performing a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the two-dimensional projection coordinates to obtain a second three-dimensional world coordinate of the first corresponding pixel point.

[0190] The implementation process of S605 can refer to the description of two-dimensional image coordinates-three-dimensional world coordinates in step S602. Specifically, the GPU can obtain the depth image Depth1 of the next frame image, and obtain the depth value Depth1(u1,v1) corresponding to the first pixel in Depth1. After that, the two-dimensional projection coordinates are multiplied by 2.0, and the difference of 1.0 is subtracted as the first two values ​​expressed as the three-dimensional space coordinates in the form of a four-dimensional array. The depth value Depth1(u1,v1) is used as the third value of the four-dimensional array, and 1.0 is used as the fourth value to fill the four-dimensional array. Then, the three-dimensional space coordinates in the form of the four-dimensional array are multiplied by the inverse matrix of the product of the world matrix matrix of the next frame image and the projection matrix to obtain a three-dimensional coordinate matrix, and then the three-dimensional coordinate matrix is ​​divided by the w component in the clipping coordinate system to obtain the three-dimensional world coordinates worldPos2.xyz of the pixel point of the first pixel in the next frame image in the world three-dimensional coordinate system, so as to realize the coordinate transformation from two-dimensional image coordinates to three-dimensional world coordinates. Optionally, the following formula or a variation of the formula can be used for implementation:

[0191] tempData=[(u1,v1)*2.0-1.0,Depth1(u1,v1),1.0]*Inverse(viewMatrix2*projectionMatrix);

[0192] worldPos2.xyz=(tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w);

[0193] Among them, dividing by tempData.w means dividing by the w component in the clipping coordinate system; tempData represents three-dimensional coordinates, x, y and z represent the parameters of the three directions of the x, y and z three-dimensional space respectively; Inverse means performing an inverse operation on the product of the projection matrix and the world matrix, that is, the inverse matrix.

[0194] S606A: If the distance between the first 3D world coordinate and the second 3D world coordinate is greater than or equal to a preset distance threshold, the first pixel point is a dynamic pixel point.

[0195] S606B: If the distance between the first 3D world coordinate and the second 3D world coordinate is less than a preset distance threshold, the first pixel point is a static pixel point.

[0196] The GPU determines whether the distance between the first 3D world coordinate and the second 3D world coordinate is greater than or equal to a preset distance threshold. If so, it means that the position of the first pixel has moved a lot and can be determined as a dynamic pixel. If less than the preset distance threshold, it means that the position of the first pixel has moved little and can be determined as a static pixel.

[0197] That is, if the length of the vector (worldPos2.xyz-worldPos1.xyz) is less than the preset distance threshold, the pixel point (u0, v0) is a static pixel point and can be marked as 0 in the Mask image; if the length of the vector (worldPos2.xyz-worldPos1.xyz) is greater than or equal to the preset distance threshold, the pixel point (u0, v0) is a dynamic pixel point and can be marked as 1 in the Mask image. It should be noted that the above preset distance threshold can be a smaller value such as 0.5 or 0.3.

[0198] If the target image is a subsequent frame image, the operation may be performed referring to the operation on the previous frame image, which will not be described in detail here.

[0199] Using the above Figure 6 In the process of separating dynamic pixels and static pixels by using the method, the two-dimensional image coordinates of the first pixel are converted into a three-dimensional world coordinate system combined with depth information, and then the three-dimensional projection coordinates of the pixel are calculated using a world matrix containing rotation parameters and translation parameters, so that the obtained coordinates have high accuracy. The two-dimensional projection coordinates are extracted based on the high-precision three-dimensional projection coordinates. When the two-dimensional projection coordinates are converted into the three-dimensional world coordinates corresponding to the first frame of the image, the depth information of another frame of the image is combined, and the accuracy of the obtained three-dimensional world coordinates is also high. Based on this, the accuracy of the separated static pixels and dynamic pixels is high, which makes the accuracy of the motion vector also high.

[0200] The above steps complete the separation of dynamic pixels and static pixels. Next, the acquisition of motion vectors of static pixels and dynamic pixels will be described in detail.

[0201] Figure 7 A flowchart of a static vector acquisition operation method provided in an embodiment of the present application includes:

[0202] S701 . Obtain first two-dimensional image coordinates of a first pixel point, where the first pixel point is a pixel point in a first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel point in the first frame image.

[0203] Here, the first frame image may be the previous frame image, and the second frame image may be the next frame image; if the first frame image is the next frame image, the second frame image may be the previous frame image. In this embodiment, the first frame image may be the previous frame image, and the second frame image may be the next frame image.

[0204] The GPU can directly obtain the two-dimensional image coordinates of a pixel point in the previous frame image, for example, obtain the two-dimensional image coordinates of the first pixel point in the previous frame image, and record them as the first two-dimensional image coordinates.

[0205] S702, performing a 2D image coordinate-3D world coordinate conversion operation on the first 2D image coordinate to obtain a first 3D world coordinate of a first pixel point;

[0206] The GPU can perform a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinates to obtain the first three-dimensional world coordinates of the first pixel point. Specifically, the GPU obtains the depth value of the first pixel point from the depth image of the previous frame image for the first two-dimensional image coordinates, and then combines the projection matrix and the world matrix of the previous frame image to convert the three-dimensional world coordinates of the first pixel point, which are recorded as the first three-dimensional world coordinates.

[0207] Optionally, the GPU can obtain the depth image Depth0 of the previous frame image, and obtain the depth value Depth0(u0,v0) of the first pixel in Depth0. After that, the two-dimensional image coordinates of the first pixel in the previous frame image are multiplied by 2.0, and the difference minus 1.0 is used as the first two values ​​of the three-dimensional space coordinates expressed in the form of a four-dimensional array. The depth value Depth0(u0,v0) is used as the third value of the four-dimensional array, and 1.0 is used as the fourth value to complete the four-dimensional array. The three-dimensional space coordinates are then multiplied by the inverse matrix of the product of the projection matrix projectionMatrix and the world matrix viewMatrix1 of the previous frame image to obtain a three-dimensional coordinate matrix, and then the three-dimensional coordinate matrix is ​​divided by the w component in the clipping coordinate system to obtain the three-dimensional world coordinates worldPos1.xyz of the first pixel in the world three-dimensional coordinate system, thereby realizing the coordinate transformation from two-dimensional image coordinates to three-dimensional world coordinates. Optionally, the following formula or a variation of the formula can be used:

[0208] tempData=[(u0,v0)*2.0-1.0,Depth0(u0,v0),1.0]*Inverse(viewMatrix1*projectionMatrix);

[0209] worldPos1.xyz=(tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w);

[0210] Among them, dividing by tempData.w means dividing by the w component in the clipping coordinate system; tempData represents three-dimensional coordinates, x, y and z represent the parameters of the three directions of the x, y and z three-dimensional space respectively; Inverse means performing an inverse operation on the product of the projection matrix and the world matrix, that is, the inverse matrix.

[0211] S703: Project the first three-dimensional world coordinates onto the second frame image to obtain three-dimensional projection coordinates.

[0212] The GPU can reproject the first 3D world coordinates generated in the previous step into the next frame image to obtain the 3D projection coordinates. Specifically, the GPU can project the first 3D world coordinates into the second frame image according to the projection matrix and the world matrix of the second frame image to obtain the 3D projection coordinates.

[0213] Optionally, the GPU may multiply the first three-dimensional world coordinates by the projection matrix projectionMatrix and the world matrix viewMatrix2 of the next frame image to obtain the three-dimensional projection coordinates.

[0214] Optionally, the following formula or a variation of the formula may be used:

[0215] tempData=[worldPos1.xyz,1.0]*viewMatrix2*projectionMatrix;

[0216] tempUVData.xyz=(tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w);

[0217] Among them, dividing by tempData.w represents the operation of dividing by the w component in the clipping coordinate system; tempData.xyz represents three-dimensional coordinates, x, y and z represent parameters in three directions of the three-dimensional space of x, y and z respectively, and * represents multiplication or generalized multiplication.

[0218] S704: Obtain a static motion vector according to the first two-dimensional image coordinates and the three-dimensional projection coordinates.

[0219] Optionally, the GPU can extract the 2D projection coordinates (u1, v1) from the 3D projection coordinates. For example, the values ​​corresponding to the x and y directions can be selected from the above tempUVData.xyz, and each of them can be added with 1.0 and then multiplied by 0.5 or a value close to 0.5 as the value of (u1, v1). It can be expressed as the following formula or a modified formula:

[0220] (u1,v1)=(tempUVData.xy+1.0)*0.5;

[0221] Among them, tempUVData.xy means that the value of (u1, v1) is the value corresponding to the x and y directions in tempUVData.xyz, and the two-dimensional projection coordinates are obtained.

[0222] Then, the GPU obtains the static motion vector of the first pixel according to the vector of the two-dimensional projection coordinates and the first two-dimensional image coordinates. If the predicted frame is located between the previous frame and the next frame, the static motion vector can refer to the following expression or a variation of the expression: [(u1-u0)*0.5, (v1-v0)*0.5]. Among them, 0.5 means taking half of the value of the vector change, which is used to determine the motion vector in the case of intermediate interpolation.

[0223] Figure 7In the method for obtaining the static motion vector of the static pixel shown in the figure, by projecting the two-dimensional image coordinates into three-dimensional world coordinates and combining the depth value of the depth image, the coordinate position is more accurate during the coordinate projection and transformation process. Afterwards, the GPU obtains a high-accuracy static motion vector based on the first two-dimensional image coordinates and the obtained precise three-dimensional projection coordinates. At the same time, since the static motion vector is directly determined based on the vector difference between the three-dimensional projection coordinates and the two-dimensional image coordinates, the amount of calculation is reduced compared to the method of determining the motion vector by combining the camera's projection matrix and the world matrix, thereby saving power consumption.

[0224] Figure 8 A flowchart of a dynamic vector acquisition operation method provided in an embodiment of the present application includes:

[0225] S801. Obtain a first brightness value of a first pixel point, where the first pixel point is any pixel point in a first frame image.

[0226] It should be noted that each pixel in a color image has a color value, also called an RGB value, which includes R, B, and G values. Taking a pixel with coordinates (u, v) in a two-dimensional image as an example, the RGB value can be expressed as: Color0(u, v).rgb, which includes R, B, and G values, which are Color0(u, v).r, Color0(u, v).g, and Color(u, v).b0, respectively. The brightness value can be obtained using the following formula or a variation of the formula:

[0227] Intensity0(u,v)=Color0(u,v).r*0.299+Color0(u,v).g*0.587+Color(u,v).b0*0.114;

[0228] Among them, Intensity0(u,v) represents the brightness value of the pixel with coordinates (u,v) in the two-dimensional image, and 0.299, 0.587 and 0.114 represent the weight coefficients of R value, G value and B value in the brightness value respectively. Optionally, the weight coefficient can also be fine-tuned; optionally, the formula can also be deformed or superimposed with correction values ​​to adapt to the needs of different scenes or converted to data of other dimensions for use, which will not be repeated here.

[0229] Specifically, the GPU uses the above method to calculate the corresponding brightness value for any pixel point in the first frame image, for example, the first pixel point. The GPU can calculate the corresponding brightness value for each pixel point.

[0230] S802: Convert the first brightness value into a first eigenvector.

[0231] Specifically, the GPU may convert the first brightness value into a first eigenvector. Optionally, the GPU may first generate a plurality of random vectors, then obtain an output value according to an offset position indicated by each random vector, and then combine the output values ​​of the offset position indicated by each random vector into a first eigenvector describing the characteristics of the first brightness value. The first eigenvector is a brightness eigenvector that ensures brightness information.

[0232] Specifically, the GPU uses the above method to obtain a corresponding feature vector for any pixel point in the first frame image. The feature vector is a brightness feature vector that ensures brightness information.

[0233] S803. Filter out a first matching pixel point that matches the first pixel point from the second pixel point set, where the second pixel point set is a set of pixels in the second frame image, a vector distance between the first matching feature vector and the first feature vector is smaller than a vector distance between other feature vectors and the first feature vector, the first matching feature vector is a feature vector converted from the brightness value of the first matching pixel point, other pixels are pixels in the second frame image other than the first matching pixel point, and other feature vectors are feature vectors converted from the brightness values ​​of other pixels.

[0234] Specifically, the GPU can perform pixel matching from the pixel points in the second frame image for any point in the first frame image, obtain the pixel point that matches the pixel point in the first frame image, and thus obtain a matching pixel point pair. Taking the first pixel point in the first frame image as an example, multiple pixel points in the second frame image are used as the second pixel point set.

[0235] The GPU can refer to S801 and S802 for each pixel in the second pixel set to obtain a feature vector generated by the brightness of each pixel. Then, the GPU calculates the vector distance between the feature vector of each pixel in the second pixel set and the first feature vector in turn, and finds the pixel corresponding to the feature vector with the smallest vector distance in the second frame image, which is recorded as the first matching pixel. The feature vector Special of the first matching pixel can be expressed by the following expression:

[0236] Special=sign(Intensity–Intensity_i)< <n|sign(Intensity–Intensity_i)<<n–1|…

[0237] Optionally, calculating the distance between two feature vectors may be calculating the Hamming distance between the two vectors.

[0238] The GPU can perform the above operation for each pixel in the first frame image, and obtain the matching pixel in the second frame image for each pixel in the first frame image, thereby obtaining a plurality of matching pixel pairs.

[0239] S804: Determine a dynamic motion vector corresponding to the first pixel point according to the difference between the two-dimensional image coordinates of the first matching pixel point in the second frame image and the two-dimensional image coordinates of the first pixel point in the first frame image.

[0240] Specifically, the GPU can obtain a dynamic motion vector based on the matching pixel pairs. Taking the pixel pair consisting of the first pixel and the first matching pixel as an example, the motion vector can be expressed as the coordinate difference between the first pixel and the first matching pixel. If the frame prediction is an interpolation between the first frame image and the second frame image, the motion vector can be expressed as the coordinate difference between the first pixel and the first matching pixel, multiplied by 0.5 or a value close to 0.5. For example, it can be expressed as the following formula or a variation of the formula:

[0241] [(u1-u0)*0.5,(v1-v0)*0.5];

[0242] The two-dimensional image coordinates of the first pixel point are (u0, v0), and the two-dimensional image coordinates of the first matching pixel point are (u1, v1).

[0243] In the above process of calculating the feature vectors one by one, the corresponding feature vector is calculated for each feature point, and the calculation process of the feature vector consumes a lot of resources. Based on this, the embodiment of the present application also provides another method for obtaining dynamic vectors. In this method, the GPU generates respective image pyramids for the first frame image and the second frame image based on the brightness values ​​of the pixels, and then selects an intermediate image of the same order and low order (i.e., an image with few pixels and small image size) from each of the two image pyramids to calculate the initial dynamic motion vector. In this process, you can refer to Figure 8In the embodiment shown, the matching pixel pair is obtained by searching the first matching pixel for the first pixel. Next, the GPU can find the local optimal solution in the higher-order image of the intermediate image where each pixel is located based on the pixel in the pixel pair, that is, find a pixel with the closest brightness value. Then, the GPU obtains the offset between the coordinates of the pixel with the closest brightness value and the original pixel on the intermediate image, and uses this offset to correct the initial dynamic motion vector obtained previously. It should be noted that such a correction method can be iterated step by step until the highest-order image. The local optimal solution in this embodiment is the pixel with the closest brightness value in the local area. In this method, the initial dynamic motion vector can be obtained by calculating a small number of feature vectors in the low-order image, and then the dynamic motion vector is corrected step by step according to the offset of the pixel of each order image until the highest-order image, thereby ensuring the accuracy of the dynamic motion vector while avoiding the budget of a large number of feature vectors. Instead, the accurate dynamic motion vector can be obtained by finding the local optimal solution of simple calculation, which reduces the amount of calculation and saves power consumption.

[0244] In order to clearly illustrate the process of iterative calculation of dynamic motion vectors using image pyramids, an exemplary description is given here with reference to the accompanying drawings.

[0245] Taking the first frame image as the previous frame image and the second frame image as the next frame image as an example, for the previous frame image, the color value of each pixel can be extracted according to the method described above to calculate the corresponding brightness value, thereby obtaining a brightness image. The brightness image can be the highest-order brightness image, that is, the bottom-level image of the image pyramid. Fig. 9 is a schematic diagram of an image pyramid. The highest-order brightness image generated here can be Fig. 9 The fifth-order image in . The number of pixels of the fifth-order image is the same as that of the original image, that is, the previous frame image.

[0246] Next, downsampling is performed based on the 5th order image. Specifically, the GPU takes the average brightness value of multiple pixels in the same area (which can be a pixel block), for example, the average brightness value of four pixels in a 2*2 pixel area, as the brightness value of the corresponding pixel in the next order brightness image, thereby generating the next order, that is, the 4th order brightness image. For example, see Fig.10 As shown, if the pixel distribution of the 5th order brightness image is Fig.10 As shown in Figure a, after downsampling, the pixel distribution of the 4th order brightness image can be seen Fig.10 As shown in Figure b. Fig.10 In the figure a, the four pixels marked with "1" are downsampled to correspond to Fig.10 The pixel marked "1" in Figure b should be noted that Fig.10 The brightness value of the pixel marked "1" in the b figure is Fig.10 The average brightness value of the four pixels marked "1" in Figure a. And, Fig.10 The pixel marked "1" in figure b and Fig.10 The relative positions of the four pixels marked "1" in the image a of are also consistent in each order of brightness image. At this time, we can define Fig.10 The pixel block composed of four pixels marked "1" in figure a is Fig.10 The pixel marked "1" in Figure b corresponds to . Similarly, Fig.10 The pixel block composed of four pixels marked "2" in figure a is Fig.10 The pixel marked “2” in Figure b corresponds to; Fig.10 The pixel block composed of four pixels marked "3", "4" and "5" in Figure a is Fig.10 The pixel points marked "3", "3", "4" and "5" in Figure b correspond to each other. It should be noted that the positions of the corresponding pixel blocks and pixel points also correspond to each other.

[0247] Based on the method of downsampling the 5th order image to obtain the 4th order image, downsampling is performed step by step until the first order to obtain the first image pyramid. It should be noted that the size of the first order image is not limited, and can be 2*2, 4*4, 8*8, etc. Of course, it can also be a larger size. If the size of the original image is large, the size of the corresponding lowest order first order image is also correspondingly large.

[0248] In this embodiment, the number of pixels and the order of the brightness image are described with smaller values ​​for convenience of description, and are not intended to limit the present case.

[0249] By adopting the above method, a corresponding second image pyramid can also be generated for the next frame image.

[0250] Exemplarily, both the first image pyramid and the second image pyramid include N-order image pyramids. Here, the brightness image in the first image pyramid is recorded as the first brightness image, and the first image pyramid includes multiple-order first brightness images; the brightness image in the second image pyramid is recorded as the second brightness image, and the second image pyramid includes multiple-order second brightness images.

[0251] The GPU can start from the lowest order brightness image to calculate the dynamic motion vector, or it can select a lower order brightness image to start. Assuming N is 12, the GPU can select a brightness image lower than the Nth order, for example, the 6th order.

[0252] Specifically, the GPU can select the 6th order first brightness image in the first image pyramid, and select the 6th order second brightness image in the second image pyramid. Pixel matching is performed on these two images to obtain multiple pixel pairs. For example, a first pixel is determined from the 6th order first brightness image, and then a pixel with the smallest vector distance to the first eigenvector of the first pixel is found in the 6th order second brightness image as the first corresponding pixel. The specific method of finding matching pixels based on brightness values ​​can be found in the previous text. Figure 8 The relevant descriptions of S801-S803 in the embodiment are not repeated here.

[0253] Afterwards, the GPU may obtain the corresponding dynamic motion vector according to the two-dimensional image coordinates of the first pixel point and the first corresponding pixel point. The specific implementation method may refer to the description of S804 above, which will not be repeated here.

[0254] Next, the GPU can search for the pixel block corresponding to the first pixel in the 7th order first brightness image, and obtain the brightness value of each pixel in the pixel block. The GPU can determine the pixel in the four pixel blocks whose brightness value is closest to the first pixel as the first intermediate pixel. The GPU can also search for the first corresponding intermediate pixel in the 7th order second brightness image according to the direction of the first intermediate pixel found in the first brightness image in the first image pyramid. For details, please refer to Fig.11 As shown, Fig.11 In figure a, the pixel 1 in the 6th order first brightness image corresponds to the pixel block composed of pixels 1-1, 1-2, 1-3 and 1-4 in the 7th order first brightness image. The pixel 1-4 has the brightness value closest to pixel 1 among 1-1, 1-2, 1-3 and 1-4. Based on the direction of 1-4 in the pixel block composed of 1-1, 1-2, 1-3 and 1-4, the GPU can determine the pixel block composed of pixels 1-1, 1-2, 1-3 and 1-4. Fig.11 In the figure b, the first corresponding pixel point in the 6th order second brightness image is 2, and the first corresponding middle pixel point in the 7th order second brightness image is 2-4.

[0255] Afterwards, the GPU may correct the initial dynamic motion vector according to the coordinates of the pixel points 1-4 and the pixel points 2-4 in their respective images to obtain an intermediate dynamic motion vector.

[0256] If the 7th-order first brightness image is not the highest-order brightness image, the GPU can repeat the above Fig.11 The relevant steps are performed, searching towards the high-order brightness image step by step, and correcting the intermediate dynamic motion vector step by step, that is, iteratively executing the above process until the highest-order brightness image is reached, and the final dynamic motion vector is obtained.

[0257] For example, Fig.11 In the figure a, the dynamic motion vector (m, v) obtained by the pixel 1 in the 6th order first brightness image and the first corresponding pixel 2 in the 6th order second brightness image is (0.5, 0.5), then the dynamic motion vector (0.5, 0.5) can be corrected to (0.52, 0.52) according to the coordinates of the pixel 1-4 closest to the pixel 1, and the (0.52, 0.52) is the intermediate dynamic motion vector. The iterative operation is performed until the final dynamic motion vector of the highest order is obtained, for example, (0.523472, 0.526874).

[0258] In order to fully describe the technical solution of this application, Fig.12 , a complete description of the vector determination method of the embodiment of the present application is given, and the specific references include:

[0259] S1201. Obtain a Hook instruction stream of a rendering resource to obtain a color image and a depth image of two frames, a projection matrix, and a world matrix.

[0260] S1202: Mark the pixels as dynamic pixels and static pixels.

[0261] The specific implementation method of this step can be found in the above Figure 6 The detailed description of the embodiments will not be repeated here.

[0262] S1203: Determine whether it is a static pixel.

[0263] Specifically, the GPU may determine one by one whether the pixels are static pixels. If they are static pixels, step S1204A may be executed. If they are dynamic pixels, step S1204B may be executed.

[0264] If it is not possible to determine whether it is a static pixel or a dynamic pixel, the process can be stopped to avoid invalid operations and release resources.

[0265] S1204A, reproject, calculate static motion vector.

[0266] The specific implementation of S1204A can be found in Figure 7 The detailed description of the embodiments will not be repeated here.

[0267] S1204B, generate an image pyramid.

[0268] S1205 , searching for the optimal solution layer by layer to obtain the final dynamic motion vector.

[0269] Searching for the optimal solution layer by layer means finding the pixel points with the closest brightness in the corresponding pixel blocks layer by layer.

[0270] For the specific implementation of S1204B and S1205, please refer to Figure 9-11 Description of the related embodiments.

[0271] Fig.12 The implementation principles and beneficial effects of each step can be found in the above description and will not be repeated here.

[0272] The above describes in detail an example of the method provided by the present application. It is understandable that, in order to implement the above functions, the corresponding device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0273] The present application can divide the functional modules of the motion vector determination device according to the above method example. For example, each function can be divided into each functional module, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0274] Fig.13 The schematic diagram of the structure of a motion vector determination device provided by the present application is shown. The device 1300 includes:

[0275] The acquisition module 1301 is used to acquire static pixel points and dynamic pixel points of the target image, where the static pixel points are pixel points whose positions in the target image have not moved, and the dynamic pixel points are pixel points whose positions in the target image have moved;

[0276] The determination module 1302 is used to perform a static vector acquisition operation on a static pixel point to obtain a static motion vector of the static pixel point; and perform a dynamic vector acquisition operation on a dynamic pixel point to obtain a dynamic motion vector of the dynamic pixel point. The motion vector includes a static motion vector and a dynamic motion vector.

[0277] In some embodiments, the acquisition module 1301 is specifically used to perform a dynamic and static pixel separation operation on each pixel in the target image to obtain a dynamic and static mask image, which is a mask image marking dynamic pixels and static pixels.

[0278] In some embodiments, the target image is a first frame image or a second frame image, and the acquisition module 1301 is specifically used to acquire first two-dimensional image coordinates of a first pixel point, where the first pixel point is a pixel point in the first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel point in the first frame image; perform a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinates to obtain first three-dimensional world coordinates of the first pixel point; project the first three-dimensional world coordinates into the second frame image to obtain three-dimensional projection coordinates; acquire two-dimensional projection coordinates according to the three-dimensional projection coordinates; perform a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the two-dimensional projection coordinates to obtain second three-dimensional world coordinates; if the distance between the first three-dimensional world coordinates and the second three-dimensional world coordinates is greater than or equal to a preset distance threshold, the first pixel point is a dynamic pixel point; if the distance between the first three-dimensional world coordinates and the second three-dimensional world coordinates is less than the preset distance threshold, the first pixel point is a static pixel point.

[0279] In some embodiments, the determination module 1302 is specifically used to obtain the first two-dimensional image coordinates of the first pixel point, where the first pixel point is the pixel point in the first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel point in the first frame image; perform a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinates to obtain the first three-dimensional world coordinates of the first pixel point; project the first three-dimensional world coordinates into the second frame image to obtain three-dimensional projection coordinates; and obtain a static motion vector based on the first two-dimensional image coordinates and the three-dimensional projection coordinates.

[0280] In some embodiments, the determination module 1302 is specifically configured to extract two-dimensional projection coordinates from three-dimensional projection coordinates; and obtain a static motion vector according to the difference between the two-dimensional projection coordinates and the first two-dimensional image coordinates.

[0281] In some embodiments, the determination module 1302 is specifically used to obtain the target three-dimensional world coordinates according to the target two-dimensional image coordinates, the depth value of the target pixel point in the target image, the projection matrix and the world matrix of the target image, where the target pixel point is the pixel point in the target image, and the target two-dimensional image coordinates are the two-dimensional image coordinates of the target pixel point in the target image; when the target image is the first frame image, the target three-dimensional world coordinates are the first three-dimensional world coordinates; when the target image is the second frame image, the target three-dimensional world coordinates are the second three-dimensional world coordinates.

[0282] In some embodiments, the determination module 1302 is specifically configured to project the first three-dimensional world coordinates into the second frame image according to the projection matrix and the world matrix of the second frame image to obtain the three-dimensional projection coordinates.

[0283] In some embodiments, the target image is a first frame image or a second frame image, and the determination module 1302 is specifically used to obtain a first brightness value of a first pixel point, where the first pixel point is any pixel point in the first frame image; convert the first brightness value into a first feature vector; screen out a first matching pixel point that matches the first pixel point from a second pixel point set, where the second pixel point set is a set of pixels in the second frame image, a vector distance between the first matching feature vector and the first feature vector is smaller than a vector distance between other feature vectors and the first feature vector, the first matching feature vector is a feature vector converted from the brightness value of the first matching pixel point, other pixels are pixels in the second frame image other than the first matching pixel point, and other feature vectors are feature vectors converted from the brightness values ​​of other pixels; determine the dynamic motion vector corresponding to the first pixel point based on the difference between the two-dimensional image coordinates of the first matching pixel point in the second frame image and the two-dimensional image coordinates of the first pixel point in the first frame image.

[0284] In some embodiments, the target image is a first frame image or a second frame image, and the determination module 1302 is specifically used to: obtain a first image pyramid, where the first image pyramid is an image pyramid generated according to the first frame image, and the first image pyramid includes an N-order first brightness image, wherein the i-th order first brightness image is a brightness image obtained by downsampling the i+1-th order first brightness image, i is a positive integer greater than 0 and less than N, the brightness value of the first downsampled pixel point in the i-th order first brightness image is the average brightness value of four pixel points corresponding to the position of the first downsampled pixel point in the i+1-th order first brightness image, and the first downsampled pixel point is any one of the i-th order first brightness image. pixel points; obtaining a second image pyramid, the second image pyramid is an image pyramid generated according to the second frame image, the second image pyramid includes an N-order second brightness image, wherein the i-th order second brightness image is a brightness image obtained by downsampling the i+1-th order second brightness image, i is a positive integer greater than 0 and less than N, and the brightness value of the second downsampled pixel point in the i-th order second brightness image and the average brightness value of four pixel points corresponding to the position of the second downsampled pixel point in the i+1-th order second brightness image, the second downsampled pixel point is any pixel point in the i-th order second brightness image; determining the j-th order first brightness image from the first image pyramid, and determining the j-th order first brightness image from the second image pyramid The j-th order second brightness image is determined in the pyramid, wherein j is a positive integer greater than 0 and less than N; according to the first pixel point and the first corresponding pixel point, an initial motion vector corresponding to the first pixel point is determined, the first pixel point is any pixel point in the j-th order first brightness image, the vector distance between the first brightness vector and the first corresponding brightness vector is less than the vector distance between the first brightness vector and other brightness vectors, the first brightness vector is a vector obtained by converting the brightness value of the first pixel point, the first corresponding brightness vector is a vector obtained by converting the brightness value of the first corresponding pixel point, other brightness vectors are vectors obtained by converting the brightness values ​​of other pixel points, and other pixel points are not the first corresponding pixel points in the j-th order second brightness image. ; obtaining an initial dynamic motion vector according to the two-dimensional image coordinates of the first pixel in the j-order first brightness image and the two-dimensional image coordinates of the first corresponding pixel in the j-order second brightness image; searching for the first intermediate pixel with the smallest brightness value difference with the first pixel in the j+1-order first brightness image; searching for the first corresponding intermediate pixel with the smallest brightness value difference with the first corresponding pixel in the j+1-order second brightness image; correcting the initial motion vector according to the two-dimensional image coordinates of the first intermediate pixel in the j+1-order first brightness image and the two-dimensional image coordinates of the first corresponding intermediate pixel in the j+1-order second brightness image to obtain an intermediate dynamic motion vector;If j is less than N-1, j+1 is used as the new j, and the step of searching for the first intermediate pixel with the smallest brightness difference from the first pixel in the first brightness image of the j+1th order is continued until j+1=N, and the dynamic motion vector is obtained. ;

[0285] In some embodiments, j is a positive integer less than half of N.

[0286] The specific manner in which the device 1300 executes the motion vector determination method and the beneficial effects produced can be found in the relevant description of the method embodiment, which will not be repeated here.

[0287] The present application also provides an electronic device, including the above processor. The electronic device provided by this embodiment can be Figure 1 The terminal device 100 shown is used to perform the above-mentioned motion vector determination method. In the case of an integrated unit, the terminal device may include a processing module, a storage module and a communication module. Among them, the processing module can be used to control and manage the actions of the terminal device, for example, it can be used to support the terminal device to execute the steps performed by the display unit, the detection unit and the processing unit. The storage module can be used to support the terminal device to execute the stored program code and data, etc. The communication module can be used to support the communication between the terminal device and other devices.

[0288] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processor (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other terminal devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0289] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device involved in this embodiment may be a Figure 1 Device of the structure shown.

[0290] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the motion vector determination method described in any of the above embodiments.

[0291] The embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the motion vector determination method in the above-mentioned embodiment.

[0292] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0293] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, the replaced units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0294] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0295] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0296] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining a motion vector, characterized in that: include: Acquire static pixel points and dynamic pixel points of a target image, wherein the static pixel points are pixel points whose positions in the target image have not moved, and the dynamic pixel points are pixel points whose positions in the target image have moved; Performing a static vector acquisition operation on the static pixel point to obtain a static motion vector of the static pixel point; A dynamic vector acquisition operation is performed on the dynamic pixel point to obtain a dynamic motion vector of the dynamic pixel point, where the motion vector includes the static motion vector and the dynamic motion vector.

2. The method according to claim 1, characterized in that The step of obtaining static pixel points and dynamic pixel points of a target image includes: A dynamic and static pixel separation operation is performed on each pixel in the target image to obtain a dynamic and static mask image, where the dynamic and static mask image is a mask image that marks dynamic pixels and static pixels.

3. The method according to claim 2, characterized in that The target image is the first frame image or the second frame image, and the moving and static pixel separation operation includes: Acquire first two-dimensional image coordinates of a first pixel point, where the first pixel point is a pixel point in the first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel point in the first frame image; Performing a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinates to obtain first three-dimensional world coordinates of the first pixel point; Projecting the first three-dimensional world coordinates onto the second frame image to obtain three-dimensional projection coordinates; Acquire two-dimensional projection coordinates according to the three-dimensional projection coordinates; Performing the two-dimensional image coordinate-three-dimensional world coordinate conversion operation on the two-dimensional projection coordinate to obtain a second three-dimensional world coordinate; If the distance between the first three-dimensional world coordinate and the second three-dimensional world coordinate is greater than or equal to a preset distance threshold, the first pixel point is the dynamic pixel point; If the distance between the first three-dimensional world coordinate and the second three-dimensional world coordinate is less than a preset distance threshold, the first pixel point is the static pixel point.

4. The method according to claim 1, characterized in that The static vector acquisition operation includes: Acquire first two-dimensional image coordinates of a first pixel point, where the first pixel point is a pixel point in a first frame image, and the first two-dimensional image coordinates are the two-dimensional image coordinates of the first pixel point in the first frame image; Performing a two-dimensional image coordinate-to-three-dimensional world coordinate conversion operation on the first two-dimensional image coordinates to obtain first three-dimensional world coordinates of the first pixel point; Projecting the first three-dimensional world coordinates onto the second frame image to obtain three-dimensional projection coordinates; The static motion vector is obtained according to the first two-dimensional image coordinates and the three-dimensional projection coordinates.

5. The method according to claim 4, characterized in that The obtaining the static motion vector according to the first two-dimensional image coordinates and the three-dimensional projection coordinates includes: Extracting two-dimensional projection coordinates from the three-dimensional projection coordinates; The static motion vector is obtained according to the difference between the two-dimensional projection coordinates and the first two-dimensional image coordinates.

6. The method according to any one of claims 3 to 5, characterized in that The two-dimensional image coordinate-three-dimensional world coordinate conversion operation includes: Obtaining target three-dimensional world coordinates according to target two-dimensional image coordinates, a depth value of a target pixel point in the target image, a projection matrix, and a world matrix of the target image, wherein the target pixel point is a pixel point in the target image, and the target two-dimensional image coordinates are the two-dimensional image coordinates of the target pixel point in the target image; When the target image is the first frame image, the target three-dimensional world coordinates are the first three-dimensional world coordinates; When the target image is the second frame image, the target three-dimensional world coordinates are second three-dimensional world coordinates.

7. The method according to claim 3 or 4, characterized in that: The step of projecting the first three-dimensional world coordinates onto the second frame image to obtain three-dimensional projection coordinates includes: According to the projection matrix and the world matrix of the second frame image, the first three-dimensional world coordinates are projected into the second frame image to obtain the three-dimensional projection coordinates.

8. The method according to claim 1, characterized in that The target image is a first frame image or a second frame image, and the dynamic vector acquisition operation includes: Obtaining a first brightness value of a first pixel, where the first pixel is any pixel in a first frame of image; Converting the first brightness value into a first eigenvector; A first matching pixel point matching the first pixel point is selected from a second pixel point set, wherein the second pixel point set is a set of pixel points in a second frame image, a vector distance between a first matching feature vector and the first feature vector is smaller than a vector distance between other feature vectors and the first feature vector, the first matching feature vector is a feature vector converted from a brightness value of the first matching pixel point, the other pixel points are pixel points in the second frame image other than the first matching pixel point, and the other feature vectors are feature vectors converted from brightness values ​​of the other pixel points; The dynamic motion vector corresponding to the first pixel point is determined based on the difference between the two-dimensional image coordinates of the first matching pixel point in the second frame image and the two-dimensional image coordinates of the first pixel point in the first frame image.

9. The method according to claim 1, characterized in that: The target image is a first frame image or a second frame image, and the dynamic vector acquisition operation includes: Acquire a first image pyramid, where the first image pyramid is an image pyramid generated according to a first frame image, and the first image pyramid includes an N-order first brightness image, wherein the i-th order first brightness image is a brightness image obtained by downsampling the i+1-th order first brightness image, i is a positive integer greater than 0 and less than N, the brightness value of a first downsampled pixel point in the i-th order first brightness image is an average brightness value of four pixel points in the i+1-th order first brightness image corresponding to the position of the first downsampled pixel point, and the first downsampled pixel point is any pixel point in the i-th order first brightness image; Acquire a second image pyramid, where the second image pyramid is an image pyramid generated according to the second frame image, and the second image pyramid includes an N-order second brightness image, where the i-th order second brightness image is a brightness image obtained by downsampling the i+1-th order second brightness image, i is a positive integer greater than 0 and less than N, and the brightness value of a second downsampled pixel point in the i-th order second brightness image and the average brightness value of four pixels corresponding to the position of the second downsampled pixel point in the i+1-th order second brightness image are obtained, and the second downsampled pixel point is any pixel point in the i-th order second brightness image; Determine a j-th order first brightness image from the first image pyramid, and determine a j-th order second brightness image from the second image pyramid, where j is a positive integer greater than 0 and less than N; Determine, according to a first pixel point and a first corresponding pixel point, an initial motion vector corresponding to the first pixel point, wherein the first pixel point is any pixel point in the j-th order first brightness image, a vector distance between a first brightness vector and a first corresponding brightness vector is less than a vector distance between the first brightness vector and other brightness vectors, the first brightness vector is a vector obtained by converting the brightness value of the first pixel point, the first corresponding brightness vector is a vector obtained by converting the brightness value of the first corresponding pixel point, the other brightness vectors are vectors obtained by converting the brightness values ​​of other pixels, and the other pixels are pixels in the j-th order second brightness image that are not the first corresponding pixel point; Obtaining an initial dynamic motion vector according to the two-dimensional image coordinates of the first pixel point in the j-th order first brightness image and the two-dimensional image coordinates of the first corresponding pixel point in the j-th order second brightness image; Searching for a first intermediate pixel point with the smallest brightness difference from the first pixel point in the j+1th order first brightness image; Searching for a first corresponding middle pixel point with the smallest brightness difference from the first corresponding pixel point in the j+1th order second brightness image; According to the two-dimensional image coordinates of the first intermediate pixel point in the j+1th order first brightness image and the two-dimensional image coordinates of the first corresponding intermediate pixel point in the j+1th order second brightness image, the initial motion vector is corrected to obtain an intermediate dynamic motion vector; If j is less than N-1, j+1 is used as the new j, and the step of searching for the first intermediate pixel point with the smallest brightness difference with the first pixel point in the j+1th order first brightness image is continued until j+1=N, and the dynamic motion vector is obtained.

10. The method according to claim 9, characterized in that The j is a positive integer less than half of N.

11. A chip, characterized in that: The chip comprises a processor; the processor is used to read and execute a computer program stored in a memory to perform the method according to any one of claims 1 to 10.

12. The chip according to claim 11, characterized in that: The chip is a graphics processing unit GPU.

13. An electronic device, characterized in that: include: Processors, memory, and interfaces; The processor, the memory and the interface cooperate with each other so that the electronic device executes the method as claimed in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 10.

Images