Motion vector determination method, chip, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of image processing, and provides a motion vector determination method, a chip, electronic equipment and a readable storage medium, and the method comprises the steps: carrying out the feature point extraction operation of a previous frame of image, and obtaining at least one first feature point; executing the feature point extraction operation on the next frame of image to obtain at least one second feature point; according to the at least one first feature point and the at least one second feature point, a rotation parameter and a translation parameter of a virtual camera are determined, and the virtual camera is used for obtaining a previous frame image and a next frame image; and determining a motion vector according to the rotation parameter and the translation parameter, the motion vector representing a motion trend of the pixel point. The method can reduce the calculation amount and save power consumption.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a motion vector determination method, chip, electronic device and readable storage medium. Background Art

[0002] In frame prediction technology, the acquisition of motion vectors is of great significance to the display effect of images. Electronic devices can determine the motion trend of pixels based on the calculated motion vectors, and realize the process of obtaining predicted images based on real images and motion vectors.

[0003] Frame prediction technology is often used in some applications (applications, APPs) with high refresh rates, such as some game APPs. In order to enhance the screen experience, game APPs have a higher screen refresh rate. In the development and update process of mobile game APPs, the screen refresh chip has developed from 60Hz to 90Hz, and now to 120Hz. Correspondingly, the graphics rendering resolution has also developed from 720p to 1080p, and even reached 2k. This will cause the GPU to have a very large amount of computation, and the performance requirements of the GPU are very high, and the power consumption is also very large. In order to reduce the GPU's computational workload and save power consumption, frame prediction technology is often used. For example, the middle frame image can be predicted based on the two real frames before and after, and the predicted image predicted by the middle frame can be interpolated, thereby ensuring that the game application has a high refresh rate while reducing the GPU's computational workload, thereby reducing power consumption. Then in the frame prediction technology, how to accurately determine the position of the pixel in the predicted image is an important factor in ensuring that the predicted image is displayed clearly and accurately.

[0004] A common method is to generate a motion vector based on the position change of each pixel in the previous frame image and the next frame image, and then move and render the pixels in the real image based on the motion vector to generate a predicted image. However, the method of generating a motion vector based on the position change of each pixel in the previous frame image and the next frame image is computationally intensive. Summary of the invention

[0005] The present application provides a motion vector determination method, device, chip, electronic device, computer-readable storage medium and computer program product, which can reduce the amount of calculation and reduce power consumption.

[0006] In a first aspect, a motion vector determination method is provided, comprising: performing a feature point extraction operation on a previous frame image to obtain at least one first feature point; performing a feature point extraction operation on a subsequent frame image to obtain at least one second feature point; determining a rotation parameter and a translation parameter of a virtual camera based on the at least one first feature point and the at least one second feature point, the virtual camera being used to obtain the previous frame image and the subsequent frame image; determining a motion vector based on the rotation parameter and the translation parameter, the motion vector representing the motion trend of a pixel point.

[0007] The GPU obtains the feature points of the previous frame and the feature points of the next frame, and then extracts the motion vector based on the matching feature points of the previous and next frames, which can greatly reduce the number of pixels to be processed, thereby reducing the amount of calculation, improving calculation efficiency, and saving power. The obtained motion vector is then applied to the global image, which can estimate the motion information of the full-screen pixels with less calculation, thereby generating a predicted image.

[0008] In one possible implementation, the feature point extraction operation includes: obtaining at least one corner point in a target image, where the target image is a previous frame image or a next frame image, and at least one corner point is a pixel point with a large brightness change in the target image; determining, among at least one corner point, a corner point whose response function value is greater than or equal to a preset response function threshold as a target feature point, where the target feature point is any one of the at least one first feature point or any one of the at least one second feature point.

[0009] By selecting corner points with large response functions as feature points, some corner points with unclear features can be further deleted, and corner points with obvious features can be retained for subsequent operations. While ensuring the transmission of motion information, the amount of calculation can be further reduced, further saving power consumption.

[0010] In a possible implementation, obtaining at least one corner point in a target image includes: obtaining a color image of the target image; and performing a corner point selection operation on each pixel in the color image to obtain at least one corner point.

[0011] The GPU can filter out corner points based on color images, which is more accurate than other image methods and can ensure the accuracy of motion vectors.

[0012] In a possible implementation, a color image carries color values ​​of multiple pixels, and a corner point selection operation includes: determining a brightness value of the first pixel according to the color value of the first pixel, where the first pixel is any one of the multiple pixels in the color image; determining brightness values ​​of N surrounding pixels according to the color values ​​of N surrounding pixels, where the N surrounding pixels are pixels distributed around the first pixel, and N is a natural number; determining a target number of surrounding pixels among the N surrounding pixels, where the absolute value of the brightness difference between the brightness value and the brightness value of the first pixel is greater than a preset brightness threshold; if the target number is greater than half of N, determining that the first pixel is a corner point; if the target number is less than or equal to half of N, determining that the first pixel is not a corner point.

[0013] The GPU can obtain the brightness value of each pixel based on the color image, and then filter out the corner points based on the difference between the brightness value of the pixel and the brightness values ​​of the surrounding pixels. It can accurately delete a large number of non-corner pixels, reducing the subsequent amount of calculations, saving power consumption, and ensuring the accuracy of the motion vector.

[0014] In one possible implementation, before determining that a corner point among at least one corner point whose response function value is greater than a preset response function threshold is a target feature point, the method includes: obtaining a brightness difference of a first corner point, where the first corner point is any one of the at least one corner point; obtaining a depth difference of the first corner point; and determining a response function corresponding to the first corner point based on the depth difference and the brightness difference.

[0015] The response function determined by combining depth difference and brightness difference can accurately combine the changes in depth and brightness, ensuring that the response function accurately expresses the characteristic changes of the corner points, and further ensures that the feature points selected based on the response function are representative, thereby ensuring the accuracy of the motion vector.

[0016] In a possible implementation, the brightness difference includes a first-direction brightness difference and a second-direction brightness difference, the first-direction brightness difference being the difference in brightness values ​​of two pixels on both sides of a first corner point in the first direction, and the second-direction brightness difference being the difference in brightness values ​​of two pixels on both sides of the first corner point in the second direction; the depth difference includes a first-direction depth difference and a second-direction depth difference, the first-direction depth difference being the difference in depth values ​​of two pixels on both sides of the first corner point in the first direction, and the second-direction depth difference being the difference in depth values ​​of two pixels on both sides of the first corner point in the second direction.

[0017] The GPU selects the brightness difference of pixels on both sides of the corner point in two different directions to determine the brightness difference, and selects the depth difference of pixels on both sides of the corner point in two different directions. This can accurately express the characteristics of the brightness change and depth change of the focus and ensure the accuracy of the motion vector.

[0018] In a possible implementation, the response function corresponding to the first corner point includes an exponential function with a natural constant e as a base and a first exponent as an exponent, and the first exponent is a function including a brightness difference and a depth difference.

[0019] The GPU uses the function of brightness difference and depth difference as the first exponent of e, and the first exponent of e as the response function of the corresponding corner point. The obtained response function is strongly correlated with the brightness difference and depth difference. Therefore, the value of the response function can accurately reflect the characteristics of the corner point and ensure the accuracy of the motion vector.

[0020] In a possible implementation, the rotation parameters and translation parameters of the virtual camera are determined according to at least one first feature point and at least one second feature point, including: determining a spatial transformation matrix according to the at least one first feature point and at least one second feature point, the spatial transformation matrix representing the change in the spatial position of the same pixel point of a subsequent frame image relative to a previous frame image; performing singular value decomposition on the spatial transformation matrix to obtain rotation parameters and translation parameters.

[0021] The GPU generates a spatial transformation matrix based on feature points. Compared with the method of generating a spatial transformation matrix based on all pixels, it can greatly reduce the number of pixels to be processed, thereby reducing the amount of calculation, improving calculation efficiency, and saving power consumption. The GPU obtains rotation parameters and translation parameters by performing singular value decomposition on the spatial transformation matrix. When the rotation parameters and translation parameters cannot be directly obtained, the GPU can also calculate the rotation parameters and translation parameters by itself, avoiding the situation where the method is limited in use due to lack of data. Therefore, it has a wider range of applications and more flexible application scenarios.

[0022] In one possible implementation, a spatial transformation matrix is ​​determined based on at least one first feature point and at least one second feature point, including: performing feature point matching on at least one first feature point and at least one second feature point to obtain at least one set of feature point pairs, the first feature point pair is any one of the at least one set of feature point pairs, the first feature point pair includes a first target feature point and a second target feature point, the first target feature point is any one of the at least one first feature point, and the second target feature point is a second feature point in the at least one second feature point that matches the first target feature point; determining the spatial transformation matrix based on at least one set of feature point pairs.

[0023] The GPU generates a spatial transformation matrix based on the matched feature points, and can extract the characteristics of spatial changes based on the motion change trends of the matched feature points, making the motion vector more accurate.

[0024] In a possible implementation, feature point matching is performed on at least one first feature point and at least one second feature point to obtain at least one set of feature point pairs, including: performing a feature vector conversion operation on at least one first feature point to obtain at least one first feature vector, and the at least one first feature vector corresponds to the at least one first feature point in one-to-one correspondence; performing a feature vector conversion operation on at least one second feature point to obtain at least one second feature vector, and the at least one second feature vector corresponds to the at least one second feature point in one-to-one correspondence; determining that the second feature vector with the smallest spatial distance from the first target feature vector in the at least one second feature vector is the second target feature vector, the first target feature vector is the first feature vector corresponding to the first target feature point, the second target feature vector is the second feature vector corresponding to the second target feature point, and the feature point pair formed by the first target feature point and the second target feature point is a pair in at least one set of feature point pairs.

[0025] The GPU converts feature points into feature vectors and then matches the feature points based on the feature distance between the feature vectors. This method has a high matching degree and makes the determined motion vector more accurate.

[0026] In a possible implementation, the feature vector conversion operation includes: obtaining multiple random vectors; extracting a first offset point and a second offset point based on a first random vector, the first random vector being any one of the multiple random vectors; obtaining a brightness value of the first offset point and a brightness value of the second offset point; outputting a first vector parameter based on a magnitude relationship between the brightness value of the first offset point and the brightness value of the second offset point; when the brightness value of the first offset point is greater than the brightness value of the second offset point, the first vector parameter is 1; when the brightness value of the first offset point is less than the brightness value of the second offset point, the first vector parameter is 0; the first vector parameter is a parameter in a first feature vector, the first feature vector also includes other vector parameters, the other vector parameters are vector parameters determined based on other random vectors, and the other random vectors are random vectors among multiple random vectors.

[0027] The GPU first generates multiple random vectors, then extracts two offset points based on each random vector, and outputs the vector parameters corresponding to the random vectors according to the size relationship of the brightness values ​​of the offset points, thereby obtaining the entire feature vector, which can accurately express the characteristics of the feature points, making the determined motion vector more accurate.

[0028] In a possible implementation, a spatial transformation matrix is ​​determined based on at least one set of feature point pairs, including: obtaining a first centroid coordinate of at least one first feature point; obtaining a first centroid offset of a first target feature point relative to the first centroid coordinate, where the first target feature point is any one of the at least one first feature point; obtaining a second centroid coordinate of at least one second feature point; obtaining a second centroid offset of the second target feature point relative to the second centroid coordinate, where the second target feature point is a second feature point in the at least one second feature point that matches the first target feature point, and a feature point pair formed by the first target feature point and the second target feature point is any pair in at least one set of feature point pairs; and multiplying each of a plurality of directional offset values ​​in the first centroid offset by each of a plurality of directional offset values ​​in the second centroid offset to obtain a spatial transformation matrix.

[0029] By obtaining the offset of the centroid coordinates of the feature points and then generating a spatial transformation matrix based on these offsets, the information of the feature points can be combined to accurately express the relative position changes between the feature points and ensure the accuracy of the motion vector.

[0030] In a possible implementation, a motion vector is determined according to a rotation parameter and a translation parameter, including: obtaining the two-dimensional image coordinates of a reference feature point, where the reference feature point is any one of at least one first feature point or at least one second feature point; determining the three-dimensional camera coordinates of the reference feature point in the camera space according to a depth image of a target image where the reference feature point is located and a projection matrix of the target image; predicting the predicted three-dimensional camera coordinates of the reference feature point according to the three-dimensional camera coordinates of the reference feature point in combination with the rotation parameter and the translation parameter; converting the predicted three-dimensional camera coordinates into predicted two-dimensional image coordinates according to the world matrix and the projection matrix of the virtual camera; and determining the motion vector according to the two-dimensional image coordinates of the reference feature point and the predicted two-dimensional image coordinates.

[0031] The GPU can first change the two-dimensional image coordinates to three-dimensional space coordinates for the feature points based on the obtained rotation parameters and translation parameters, thereby combining the depth information of the depth value of the pixel point, and calculating the predicted three-dimensional space coordinates of the pixel point in the intermediate prediction frame based on the rotation parameters and translation parameters in the three-dimensional space, and the predicted coordinates are more accurate. Finally, the predicted two-dimensional image coordinates of the pixel point in the prediction frame are obtained through the coordinate change from the three-dimensional space coordinates to the two-dimensional image coordinates, and then the motion vector is obtained based on the coordinate change from the two-dimensional image coordinates of the feature point to the predicted two-dimensional image coordinates. This method combines the depth information for rotation and translation, and the obtained motion information has high accuracy, so the obtained motion vector has high accuracy.

[0032] In a possible implementation manner, the reference feature point is any one of the at least one first feature point.

[0033] That is to say, determining the motion vector based on the previous frame image as a reference image is in line with people's habits and is easy to understand and operate.

[0034] In a second aspect, a motion vector determination device is provided, comprising a unit composed of software and / or hardware, wherein the unit is used to execute any one of the methods in the technical solution of the first aspect.

[0035] In a third aspect, an embodiment of the present application provides a chip, comprising a processor; the processor is used to read and execute a computer program stored in a memory to execute any one of the methods in the technical solution described in the first aspect.

[0036] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0037] Optionally, the chip is a graphics processing unit GPU.

[0038] Further optionally, the chip also includes a communication interface.

[0039] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor, a memory and an interface; the processor, the memory and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution described in the first aspect.

[0040] In a fifth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising any chip in the technical solution as described in the third aspect.

[0041] In a sixth aspect, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution described in the first aspect.

[0042] In a seventh aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is run on an electronic device, the electronic device executes any one of the methods in the technical solution described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of the structure of a terminal device 100 provided in an embodiment of the present application;

[0044] Figure 2 is a software structure block diagram of a terminal device provided in an embodiment of the present application;

[0045] Figure 3 This is an example of a game video frame rendering and display process provided by an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of various coordinate systems provided in an embodiment of the present application;

[0047] Figure 5 is a flowchart of a method for determining a motion vector provided in an embodiment of the present application;

[0048] Figure 6 is a flow chart of a method for determining a motion vector according to a rotation parameter and a translation parameter provided in an embodiment of the present application;

[0049] Figure 7 This is a flowchart of a corner point selection operation method provided in an embodiment of the present application;

[0050] Figure 8 is a schematic diagram of an example of a pixel selection method provided in an embodiment of the present application;

[0051] Fig. 9 is a flowchart of a method for obtaining a response function provided in an embodiment of the present application;

[0052] Fig.10 is a flowchart of a method for feature point matching provided in an embodiment of the present application;

[0053] Fig.11 is a schematic diagram of an example of a marking method for different threads provided in an embodiment of the present application;

[0054] Fig.12 is a flow chart of a method for obtaining a spatial transformation matrix provided in an embodiment of the present application;

[0055] Fig.13 is a flowchart of another method for determining a motion vector provided in an embodiment of the present application;

[0056] Fig.14 It is a schematic diagram of the structure of a motion vector determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0058] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0059] The motion vector determination method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0060] For example, Figure 1 1 is a schematic diagram of the structure of an example of a terminal device 100 provided in an embodiment of the present application. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0063] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0064] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0067] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0068] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0069] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0070] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0071] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0072] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0073] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0074] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.

[0075] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0076] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0077] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 1 The structures of antenna 1 and antenna 2 in the figure are only an example. Each antenna in the terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0078] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0079] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be arranged in the same device as the mobile communication module 150 or other functional modules.

[0080] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0081] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a BeiDou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0082] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0083] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0084] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0085] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0086] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0087] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0088] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0089] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0090] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0091] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0092] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0093] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0094] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0095] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0096] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0097] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0098] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0099] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0100] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0101] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0102] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.

[0103] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0104] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0105] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0106] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0107] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0108] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0109] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0110] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0111] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0112] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0113] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0114] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0115] Figure 2 1 is a software structure diagram of the terminal device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0116] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0117] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0118] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0119] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0120] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0121] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0122] The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (including connection, disconnection, etc.).

[0123] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0124] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0125] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0126] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0127] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0128] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0129] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0130] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0131] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0132] A 2D graphics engine is a drawing engine for 2D drawings.

[0133] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0134] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 1 and Figure 2 Taking the terminal device with the shown structure as an example, the motion vector determination method provided in the embodiment of the present application is specifically explained in combination with the accompanying drawings and application scenarios.

[0135] Motion vectors are usually used in frame prediction technology. Electronic devices can determine the motion trend of pixels based on the calculated motion vectors, and implement the process of obtaining predicted images based on real images and motion vectors. Frame prediction technology is often used in some applications (applications, APPs) with high refresh rates, such as some game APPs. In order to enhance the picture experience, game APPs have a higher screen refresh rate. In the development and update process of mobile game APPs, the screen refresh chip has developed from 60Hz to 90Hz, and now to 120Hz. Correspondingly, the graphics rendering resolution has also developed from 720p to 1080p, and even reached 2k. This will cause the GPU to have a very large amount of computation, and the performance requirements of the GPU are very high, and the power consumption is also very large. In order to reduce the GPU's computational workload and save power consumption, frame prediction technology is often used. For example, the middle frame image can be predicted based on the previous and next two frames of real images, and the predicted image predicted by the middle frame can be interpolated, thereby ensuring that the game application has a high refresh rate while reducing the GPU's computational workload, thereby reducing power consumption.

[0136] In frame prediction technology, how to accurately determine the position of pixels in the predicted image is an important factor in ensuring that the predicted image is displayed clearly and accurately. A common method is to generate a motion vector based on the position change of each pixel in the previous frame image and the next frame image, and then move and render the pixels in the real image based on the motion vector to generate a predicted image, but this method is computationally intensive.

[0137] In the technical solution of the present application, by obtaining the feature points of the previous frame image and the feature points of the next frame image, and then extracting the motion vector based on the matched feature points of the previous and next frames, the number of pixels to be processed can be greatly reduced, thereby reducing the amount of calculation, improving the calculation efficiency, and saving power consumption. The obtained motion vector is then applied to the global image, and the motion information of the full-screen pixels can be estimated with less calculation, thereby generating a predicted image.

[0138] The technical solution of the embodiment of the present application can be used in the AR / VR field, and can also be used in the game application field. Next, the application in the frame prediction of the game video played by the game application is used as an example for explanation. Figure 3 The rendering and display process of the game video frame provided by the embodiment of the present application is shown.

[0139] The game application runs on the operating system of the terminal device. In the game scene, the game application modifies the function pointer of the rendering API to generate the game's rendering instruction stream, such as hooking the game's rendering instruction stream. The game's rendering instruction stream is passed to the GPU through the graphics manager and GPU driver in turn. The GPU obtains the function through the rendering instruction and identifies the game's rendering instruction stream based on the characteristics of the game's rendering instruction.

[0140] The rendering instruction stream is an instruction stream composed of multiple drawing instructions (rendering instruction sequence). Generally, a rendering instruction stream is used to draw a frame of image. Specifically, the GPU can execute each drawing instruction in the rendering instruction stream to realize the drawing of each object in a frame of image. The set of objects generated according to the rendering instruction stream can form a frame of image. A frame of image is a frame of static image displayed by the game application. Each frame of static image can be composed of 3D scene image and UI image, etc., wherein the 3D scene image is also called color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects and skills, etc., and the UI image may include control buttons (such as running buttons and skill buttons, etc.), small maps, floating text, user avatars, energy pack controls and blood bars, etc., and the UI image is located on the upper layer of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image belong to the objects in the game frame. In addition, through the rendering instruction stream, the world matrix and projection matrix of the virtual camera corresponding to a frame of image, as well as general information such as the depth image corresponding to a frame of image can also be obtained.

[0141] The game frames in the embodiment of the present application can be divided into two types:

[0142] One is the real frame generated according to the rendering instruction stream, which is the real image mentioned above. In the game scene, the operating system sends rendering instruction streams to the GPU in sequence. The GPU generates a real frame corresponding to each rendering instruction stream in sequence according to the order in which these rendering instruction streams are received. Among them, one rendering instruction stream is used to draw one real frame.

[0143] The other is a predicted frame generated based on two rendering instruction streams obtained successively, which is the predicted image mentioned above. "Two rendering instruction streams obtained successively" means obtaining rendering instruction stream 1 first, and then obtaining rendering instruction stream 2. Specifically, the GPU can obtain two rendering instruction streams successively (one rendering instruction stream is used to draw a real frame), and separate the 3D scene image and UI image of two adjacent frames from each rendering instruction stream, as well as obtain the depth image corresponding to each real frame, the world matrix and projection matrix of the game virtual camera and other general information, and then use this information to calculate the motion vector of each pixel, and then perform pixel-by-pixel offset on the 3D scene image corresponding to a real frame based on the motion vector. Afterwards, the GPU can call the neural network model of the NPU to perform image repair on the blank area after the pixel-by-pixel offset, and superimpose the repaired 3D scene image with the UI image corresponding to this real frame to generate the final predicted frame.

[0144] Assuming that the GPU generates real frame N-1 according to the rendering instruction stream 1 obtained first, generates real frame N+1 according to the rendering instruction stream 2 obtained later, and generates predicted frame N according to rendering instruction streams 1 and 2, the GPU can send these frames to the queue to be displayed in the order of real frame N-1, predicted frame N and real frame N+1. Then the display screen plays real frame N-1, predicted frame N and real frame N+1 in sequence.

[0145] It should be understood that since the resource consumption of generating predicted frames using inter-frame prediction technology is less than the resource consumption of rendering real frames by game applications, by reducing the number of real frames generated by game applications and using inter-frame prediction technology to generate predicted frames, not only the total amount of game frames can be guaranteed, but also the power consumption load of the chip can be reduced.

[0146] For example, in a game scenario, the GPU can continuously obtain a rendering instruction stream from the game application. Whenever a rendering instruction stream is obtained, the GPU can perform the following steps 1 and 2 for the rendering instruction stream.

[0147] Step 1: The GPU separates the 3D scene image and the UI image from the rendering instruction stream.

[0148] The 3D scene image mentioned above refers to a color image that does not contain UI pixels but contains a 3D scene, that is, an RGB image.

[0149] For example, the above-mentioned rendering instruction stream may be a Hook game rendering instruction stream. The GPU may identify the various instructions contained in the Hook game rendering instruction stream through the rendering instruction acquisition function according to the characteristics of the game's rendering instruction stream. The Hook game rendering instruction stream generally includes a variety of drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Among them, the scene drawing instructions can be used to draw 3D scene images such as scenery, characters, special effects, and skills in the game, the UI drawing instructions can be used to draw UI images such as control buttons, small maps, and floating text, and the image display instructions are used to put the drawn image data into the specified location of the operating system (such as framebuffer0) to complete the frame display.

[0150] The pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located on the upper layer of the 3D scene image. If the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, the object in the UI image may cause visual occlusion to the object in the 3D scene image. The object in the 3D scene image may change its pixel coordinates for two reasons: one reason is that the position of objects such as characters in the game scene moves; the other reason is that the virtual camera in the game scene shakes, that is, the character's perspective changes. When the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, regardless of the above reasons for changing the pixel coordinates, if the object in the 3D scene image is moved according to the following steps 3-5, the object in the UI image will move with the object in the 3D scene image, but in fact the object in the UI image may not move, or the motion vectors of the object in the UI image and the object in the 3D scene image are different, which will cause the final predicted frame to have a more serious image quality problem, so the GPU needs to separate the 3D scene image and the UI image from the rendering instruction stream.

[0151] Specifically, the GPU can identify whether it is a UI drawing instruction based on the characteristics of the game rendering instruction. For example, the UI drawing instruction may include resource binding instructions, state setting instructions, and drawing instructions, and point the output of these instructions to a newly created image. The newly created image can be used to save UI pixels, so the newly created image can be called a UI image. In addition, the GPU can change the original rendering generation result to a newly created image for saving a 3D scene image that does not contain UI pixels. In this way, the GPU separates the UI image and the 3D scene image from the rendering instruction stream.

[0152] Step 2: The GPU obtains the depth image, the world matrix and the projection matrix of the virtual camera according to the rendering instruction stream.

[0153] The GPU can identify the rendering instruction sequence for model rendering based on the characteristics of the game rendering instructions, and determine the glBindBuffer instruction located before the model rendering drawcall instruction. Among them, the glBindBuffer instruction contains the relevant data of the virtual camera. The GPU can obtain the world matrix and projection matrix of the virtual camera from the buffer corresponding to the glBindBuffer instruction. In addition, the GPU can also obtain the depth image from the rendering instruction stream.

[0154] It should be noted that the 3D scene image obtained by the GPU from step 1 and the depth image obtained from step 2 have the same image content, for example, both images contain people, trees, and mountains. The difference between the two images is that the 3D scene image is a three-channel (RGB) color image; the depth image is a single-channel image, which is similar to a grayscale image. Each pixel value of the depth image can be used to represent the distance between the virtual camera and the object, and the depth image reflects the geometric shape of the visible surface of the object.

[0155] In order to facilitate the understanding of the world matrix and projection matrix of the above virtual camera, the following Figure 4 The pixel coordinate system, image coordinate system, normalized device coordinate system (NDC), camera coordinate system and world coordinate system involved in the embodiments of the present application are explained.

[0156] The pixel coordinate system o-wh and the image coordinate system o-xy are two-dimensional coordinate systems. Usually, images are stored in the form of arrays in terminal devices, and the value of each pixel in the array represents the brightness of the image point. The origin of the pixel coordinate system o-wh is located in the upper left corner of the image, the w axis is horizontal to the right, and the h axis is vertically downward. Each pixel coordinate (w,h) is the number of columns and rows of a pixel in the array, so (w,h) is the coordinate of the pixel coordinate system o-wh in pixels. The origin of the image coordinate system o-xy is located at the center of the image (called the principal point of the image), the x axis is parallel to the w axis, and the y axis is parallel to the h axis. The coordinates in the pixel coordinate system o-wh and the image coordinate system o-xy can be converted to each other.

[0157] The standardized device coordinate system (NDC coordinate system) o-x'y'z' is a three-dimensional coordinate system. In the standardized device coordinate system o-x'y'z', the values ​​of x', y' and z' are in the interval [-1,1], where z' represents the depth value of the image, and x' and y' are obtained by normalizing the coordinates (x, y) in the image coordinate system o-xy, respectively. Through the viewport transform, the standardized device coordinate system o-x'y'z' can be transformed into the image coordinate system o-xy or the pixel coordinate system o-wh. Taking a frame of 512*512 size image as an example, the pixel coordinates of the lower right corner point in the pixel coordinate system o-wh are (512, 512). After the coordinate transformation, the coordinates of the point in the standardized device coordinate system o-x'y'z' are (1, 1, z'), where z' represents the depth value of the coordinate. Optionally, z' can also be a normalized depth value. Converting the image coordinate system into a standardized coordinate system is equivalent to normalization.

[0158] The camera coordinate system O0-X0Y0Z0 is a three-dimensional coordinate system. In the camera coordinate system O0-X0Y0Z0, the virtual camera is a virtual camera set for the game scene, not a real physical camera. In the game scene, if there are multiple players roaming in the game world, since each player has a different position and sight direction in the game world, the frame displayed by the terminal device used by each player is also different. Suppose a coordinate system is created with a certain player as the origin. This coordinate system can be called the camera coordinate system, the observation space coordinate system or the three-dimensional camera coordinate system. Figure 4 , the origin O0 of the camera coordinate system is the optical center (projection center) of the virtual camera, the X0 axis and the Y0 axis are parallel to the x-axis and y-axis of the camera plane, the Z0 axis is the optical axis of the virtual camera, the Z0 axis is perpendicular to the camera plane, and oO0 is the focal length of the virtual camera. It should be understood that each player has a different position and sight direction in the game world, so the camera coordinate system O0-X0Y0Z0 established with the virtual camera corresponding to each player as the center will also be different.

[0159] The world coordinate system O-XYZ is a three-dimensional coordinate system introduced to describe the position of each virtual camera. In the game scene, virtual objects such as characters, tables, rooms, animals and vehicles have a specific position in the game. The position of the virtual object in the game world is defined as the coordinate in the world coordinate system, which is called the world space coordinate (also called the three-dimensional world coordinate). For example, the world coordinate system O-XYZ can be a three-dimensional coordinate system established with the center of the earth as the origin. It should be understood that different players have different positions and sight directions in the world coordinate system O-XYZ, which makes the position and orientation of the virtual camera in the world coordinate system O-XYZ also different. The coordinates in the camera coordinate system O0-X0Y0Z0 can be converted into coordinates in the world coordinate system O-XYZ through translation and rotation.

[0160] The world matrix (viewMatrix), also known as the external parameter matrix, is used to convert the object coordinates in the world coordinate system O-XYZ into the camera coordinate system O0-X0Y0Z0. That is, when the perspective of the virtual camera is used as the origin of the game scene, the world matrix transforms all world space coordinates into observation coordinates relative to the position and direction of the virtual camera.

[0161] The projection matrix (projectionMatrix), also known as the intrinsic parameter matrix, is used to convert the object coordinates in the camera coordinate system O0-X0Y0Z0 into descriptions using the standardized device coordinate system o-x'y'z'. When rendering, a three-dimensional scene is usually projected onto the screen (i.e., the camera surface) to become a two-dimensional image. Specifically, all vertex data is first converted from the camera coordinate system O0-X0Y0Z0 to the clipping space, and then the converted clipping coordinates are transformed into the standardized device coordinate system o-x'y'z' by dividing by the w component in the clipping coordinate system. Among them, the camera coordinate system O0-X0Y0Z0 is a right-handed coordinate system, and the standardized device coordinate system o-x'y'z' is a left-handed coordinate system. The vertex data is converted from the camera coordinate system to the clipping space coordinate system to achieve linear transformation and projection transformation, and then divided by the w component in the clipping coordinate system to achieve the conversion from homogeneous coordinates to non-homogeneous coordinates.

[0162] Assume that the GPU obtains rendering instruction stream 1 and rendering instruction stream 2 in sequence. Among them, rendering instruction stream 1 can be used to generate real frame N-1, and rendering instruction stream 2 can be used to generate real frame N+1. According to the above steps 1 and 2, the GPU can obtain the 3D scene image Color0, UI image UI0, depth image Depth0, virtual camera's world matrix ViewMatrix0 and virtual camera's projection matrix ProjectionMatrix0 corresponding to the real frame N-1, and the 3D scene image Color1, UI image UI1, depth image Depth1, virtual camera's world matrix ViewMatrix1 and virtual camera's projection matrix ProjectionMatrix1 corresponding to the real frame N+1. Then, the GPU can obtain the motion vector according to the following method.

[0163] Next, how the present application determines the motion vector is described in detail with reference to the accompanying drawings. It should be noted that the motion vector can be used for various business needs such as image editing, image drawing, and image restoration, and the present application embodiment does not limit this. The executor of the technical solution of the present application embodiment can be a terminal device, a GPU in the terminal device, or other software or hardware functional modules in the terminal device.

[0164] Specifically, Figure 5 A flowchart of a method for determining a motion vector provided in an embodiment of the present application may specifically include:

[0165] S501: Perform a feature point extraction operation on a previous frame image to obtain at least one first feature point.

[0166] S502: Perform a feature point extraction operation on the next frame image to obtain at least one second feature point.

[0167] The previous frame image can be considered as the N-1 frame image, and the next frame image can be considered as the N+1 frame image, both of which are real images, that is, images calculated by the game application. Specifically, after obtaining the previous frame image and the next frame image, the GPU performs a feature point extraction operation on the previous frame image, thereby extracting one or more first feature points. The GPU also performs a feature point extraction operation on the next frame image to obtain one or more second feature points. Usually, the extracted feature points are pixel points with strong spatial or color change information in the image.

[0168] Optionally, the feature point extraction operation can be performed by artificial intelligence, or by first extracting corner points and then extracting feature points from the corner points as mentioned below, which is not limited in this example.

[0169] S503: Determine a rotation parameter and a translation parameter of a virtual camera according to at least one first feature point and at least one second feature point, where the virtual camera is used to acquire a previous frame image and a next frame image.

[0170] Specifically, the GPU can determine the rotation parameter (rotation) and translation parameter (translation) of the virtual camera according to the relationship between at least one first feature point extracted from the previous frame image and at least one second feature point extracted from the next frame image. It should be noted that the rotation parameter is used to characterize the posture change of the virtual camera for obtaining the previous frame image and the next frame image, and the translation parameter is used to characterize the position change of the virtual camera for obtaining the previous frame image and the next frame image.

[0171] Optionally, the GPU may also determine the number of the first feature point and the second feature point. If the number of the first feature point and / or the second feature point is too small, then a small number of feature points may not be able to fully express the overall motion trend of the global image. The GPU may execute S503 only when the number of the first feature point and the second feature point is greater than or equal to the preset feature point number threshold. If the number of the first feature point and / or the second feature point is less than the preset feature point number threshold, the process is stopped and no longer processed; or the image whose number of the first feature point and the second feature point is less than the preset feature point number threshold is re-identified, so as to obtain a sufficient number of feature points, and continue to execute the subsequent process. Optionally, the preset feature point number threshold may be a natural number such as 8, 10 or 15, which is not limited in the embodiment of the present application. It should be noted that the larger the preset feature point number threshold, the more feature points there are, the more accurate the motion vector obtained, the smaller the preset feature point number threshold, the fewer feature points obtained, the less computational effort, and the more power consumption is saved. Specifically, the value can be balanced between accuracy and power consumption.

[0172] S504: Determine a motion vector according to the rotation parameter and the translation parameter, where the motion vector represents the motion trend of the pixel.

[0173] Specifically, based on a real frame, the GPU can predict the 2D image coordinates of each pixel on the real frame by combining the rotation parameter and the translation parameter. Then the vector of the 2D image coordinates of the pixel on the predicted frame pointing to the 2D image coordinates on the real frame represents the motion vector of the pixel, that is, the 2D image coordinates of the pixel on the real frame minus the 2D image coordinates on the predicted frame can represent the motion vector of the pixel.

[0174] Optionally, the GPU can predict the two-dimensional image coordinates of the first feature point in the predicted frame based on the two-dimensional image coordinates of the first feature point in the previous frame image in combination with the rotation parameters and the translation parameters. When there are multiple first feature points, the two-dimensional image coordinates corresponding to each first feature point in the predicted frame can be obtained, thereby obtaining the motion vector corresponding to each feature point. All motion vectors of multiple first feature points can be applied to the global image to realize frame prediction of the entire image. The specific implementation method of this step can be found in the following Figure 6 The embodiments shown will not be described in detail here.

[0175] Optionally, the GPU can also predict the corresponding two-dimensional image coordinates of the second feature point in the predicted frame based on the two-dimensional image coordinates of the second feature point in the next frame image, combined with the rotation parameters and the translation parameters. When there are multiple second feature points, the corresponding two-dimensional image coordinates of each second feature point in the predicted frame can be obtained, thereby obtaining the motion vector corresponding to each feature point. All motion vectors of multiple second feature points can be applied to the global image to achieve frame prediction of the entire image.

[0176] Above Figure 5 In the embodiment shown, the GPU obtains the feature points of the previous frame image and the feature points of the next frame image, and then extracts the motion vector based on the matched feature points of the previous and next frames, which can greatly reduce the number of pixels to be processed, thereby reducing the amount of calculation, improving the calculation efficiency, and saving power consumption. The obtained motion vector is then applied to the global image, and the motion information of the full-screen pixels can be estimated with less calculation, thereby generating a predicted image.

[0177] Here, we take the previous frame image as an example to describe in detail how to determine the motion vector based on the rotation parameters and translation parameters. For details, please refer to Figure 6 As shown:

[0178] S601. Acquire two-dimensional image coordinates of a reference feature point, where the reference feature point is any one of at least one first feature point or at least one second feature point.

[0179] Here, the previous frame image is taken as an example, and the reference feature point is at least one feature point in the previous frame image, and any feature point is selected to perform coordinate transformation.

[0180] Specifically, each feature point in the previous frame image needs to execute the steps of this embodiment.

[0181] Specifically, the GPU may obtain the two-dimensional image coordinates of the reference feature point in the previous frame image, recorded as (u, v).

[0182] S602 : Determine the three-dimensional camera coordinates of the reference feature point in the camera space according to the depth image of the target image where the reference feature point is located and the projection matrix of the target image.

[0183] Specifically, the GPU can also obtain the depth image and projection matrix of the target image where the reference feature point is located. For example, the GPU obtains Depth0 of the previous frame image, obtains the depth value Depth(u,v).x of the reference feature point in Depth0, and then multiplies the two-dimensional image coordinates of the reference feature point in the target image by 2.0, and subtracts the difference of 1.0 as the first two values ​​of the three-dimensional space coordinates expressed in the form of a four-dimensional array, and uses the depth value Depth(u,v).x as the third value of the four-dimensional array, and 1.0 as the fourth value to complete the four-dimensional array, and then multiplies the three-dimensional space coordinates in the four-dimensional array by the inverse matrix of the projection matrix of the target image to obtain a three-dimensional coordinate matrix, and then divides the three-dimensional coordinate matrix by the w component in the clipping coordinate system to obtain the three-dimensional camera coordinate viewPos.xyz of the reference feature point in the camera space, thereby realizing the coordinate transformation from two-dimensional to three-dimensional space.

[0184] For example, the following formula or its variation can be used:

[0185] tempData=[(u,v)*2.0–1.0,Depth(u,v).x,1.0]*Inverse(projectionMatrix);

[0186] viewPos.xyz=

[0187] (tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w).

[0188] Among them, dividing by tempData.w means dividing by the w component in the clipping coordinate system; tempData represents three-dimensional coordinates, x, y and z represent the parameters of the three directions of the x, y and z three-dimensional space respectively; Inverse means performing an inverse operation on the projection matrix, that is, the inverse matrix.

[0189] S603 , predicting the predicted 3D camera coordinates of the reference feature points according to the 3D camera coordinates of the reference feature points in combination with the rotation parameters and the translation parameters.

[0190] The rotation parameters and translation parameters obtained above express the change of the position of a pixel from the previous frame to the next frame, which can express the movement trend of the pixel and belong to motion information. Specifically, the GPU predicts the coordinates of the pixel of the benchmark feature point in the predicted frame based on the 3D camera coordinates of the benchmark feature point, combined with the rotation parameters and translation parameters, which are motion information, that is, the predicted 3D camera coordinates of the benchmark feature point.

[0191] Specifically, when the predicted frame is an image between the previous frame and the next frame, for the predicted frame, the motion of the same pixel is half of the motion represented by the corresponding rotation parameter and translation parameter. Therefore, the rotation parameter and translation parameter need to be multiplied by 0.5 or a value close to 0.5, respectively, which can be obtained by referring to the following formula or a variation of the formula:

[0192] viewPos.xyz(prediction) = viewPos.xyz*rotation*0.5+translation*0.5;

[0193] In the above formula, viewPos.xyz represents the three-dimensional camera coordinates of the reference feature point obtained in step S602 in the camera space, and viewPos.xyz(prediction) is the predicted three-dimensional camera coordinates of the predicted reference feature point.

[0194] S604: Convert the predicted three-dimensional camera coordinates into predicted two-dimensional image coordinates according to the world matrix and projection matrix of the virtual camera.

[0195] Specifically, in order for the GPU to display the predicted frame, it is also necessary to convert the predicted 3D camera coordinates into 2D coordinates in the image coordinate system, that is, project them back to the image coordinate system. The conversion of 3D coordinates to 2D coordinates can refer to the inverse process of 2D coordinate conversion to 3D. Specifically, the predicted 3D camera coordinates viewPos.xyz (prediction) are used as the first three values ​​of the 4D vector, and 1.0 is used as the fourth value to complete the 4D array, and multiplied by the projection matrix to obtain the predicted 2D image coordinates, recorded as (u_o, v_o).

[0196] For example, the following formula can be used:

[0197] tempData = (viewPos.xyz(prediction), 1.0) * projectionMatrix;

[0198] tempData.xyz=

[0199] (tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w);

[0200] Among them, dividing by tempData.w represents the operation of dividing by the w component in the clipping coordinate system; tempData.xyz represents three-dimensional coordinates, x, y and z represent parameters in three directions of the three-dimensional space of x, y and z respectively, and * represents multiplication or generalized multiplication.

[0201] Optionally, the predicted two-dimensional image coordinates (u_o, v_o) can be obtained by selecting the values ​​corresponding to the x and y directions in the above tempData.xyz, adding 0.1 to each of them and then multiplying by 0.5 or a value close to 0.5 as the value of (u_o, v_o).

[0202] It can be expressed as a formula:

[0203] (u_o,v_o)=(tempData.xy+1.0)*0.5.

[0204] Among them, tempData.xy represents the two-dimensional image coordinates obtained by selecting the values ​​corresponding to the x and y directions in tempData.xyz.

[0205] S605 . Determine a motion vector according to the two-dimensional image coordinates of the reference feature point and the predicted two-dimensional image coordinates.

[0206] Specifically, the GPU can determine the motion vector based on the two-dimensional image coordinates of the reference feature point and the predicted two-dimensional image coordinates. If the reference feature point is a feature point in the selected previous frame image, the motion vector mv can be expressed as:

[0207] mv=(u_o-u,v_o-v).

[0208] It should be noted that the GPU needs to calculate the mv for each feature point in the selected target image, thereby obtaining the mvs of multiple feature points. The mvs of these multiple feature points can be applied to the global image, that is, applied to each pixel of the entire image, to obtain the predicted frame.

[0209] Above Figure 6In the embodiment shown, the GPU can first change the two-dimensional image coordinates to three-dimensional space coordinates for the feature points based on the obtained rotation parameters and translation parameters, thereby combining the depth information of the depth value of the pixel point, and calculating the predicted three-dimensional space coordinates of the pixel point in the intermediate prediction frame based on the rotation parameters and translation parameters in the three-dimensional space, and the predicted coordinates are more accurate. Finally, the predicted two-dimensional image coordinates of the pixel point in the prediction frame are obtained through the coordinate change from the three-dimensional space coordinates to the two-dimensional image coordinates, and then the motion vector is obtained based on the coordinate change from the two-dimensional image coordinates of the feature point to the predicted two-dimensional image coordinates. This method combines the depth information for rotation and translation, and the obtained motion information has high accuracy, so the obtained motion vector has high accuracy.

[0210] Based on the above embodiments, the feature point extraction operation may include: obtaining at least one corner point in the target image, the target image is the previous frame image or the next frame image, and at least one corner point is a pixel point with large brightness change in the target image; determining that among at least one corner point, the corner point whose response function value is greater than a preset response function threshold is a target feature point, and the target feature point is any one of the at least one first feature point or any one of the at least one second feature point.

[0211] Specifically, taking the feature point extraction operation for the previous frame image as an example, the color image of the previous frame image is first obtained, and the corner point selection operation is performed on each pixel in the color image to identify the corner points in the color image. The number of corner points obtained may be one or more, and then the value of the response function of each corner point is calculated, and then the corner points with large response function values ​​are selected from the obtained corner points as the first feature points, and the corner points with small response functions are discarded.

[0212] Optionally, the above corner point selection operation is applied to the first pixel point in the color image of the previous frame image as an example, which can be seen in Figure 7 The method shown comprises:

[0213] S701. Determine a brightness value of a first pixel according to a color value of the first pixel, where the first pixel is any one of a plurality of pixels in a color image.

[0214] It should be noted that each pixel in a color image has a color value, also called an RGB value, which includes R, B, and G values. Taking a pixel with coordinates (u, v) in a two-dimensional image as an example, the RGB value can be expressed as: Color0(u, v).rgb, which includes R, B, and G values, Color0(u, v).r, Color0(u, v).g, and Color(u, v).b0, respectively. The brightness value can be obtained using the following formula or a variation of the formula:

[0215] Intensity0(u,v)=Color0(u,v).r*0.299+Color0(u,v).g*0.587+Color(u,v).b0*0.114;

[0216] Among them, Intensity0(u,v) represents the brightness value of the pixel with coordinates (u,v) in the two-dimensional image, and 0.299, 0.587 and 0.114 represent the weight coefficients of R value, G value and B value in the brightness value respectively. Optionally, the weight coefficient can also be fine-tuned; optionally, the formula can also be deformed or superimposed with correction values ​​to adapt to the needs of different scenes or converted to data of other dimensions for use, which will not be repeated here.

[0217] The GPU needs to calculate the corresponding brightness value for each pixel.

[0218] S702. Determine brightness values ​​of N surrounding pixels according to color values ​​of N surrounding pixels, where the N surrounding pixels are pixel points distributed around a first pixel, and N is a natural number.

[0219] Specifically, taking the first pixel point as an example, the GPU may select multiple pixel points around the first pixel point as surrounding pixel points.

[0220] Optionally, N can be a natural number such as 4, 6, 8, etc. For details, see Figure 8 As shown, Figure 8 In the example in which N is 4, if the first pixel is a, the surrounding pixels may include pixels 1, 2, 3, and 4. Figure 8 When selecting the surrounding pixel points in , the pixel points at the position with a difference of two pixels are selected for example. Optionally, the pixel points at the position with a difference of three pixels or the pixel points at the position with another number of pixels can also be selected. If N is 6 or 8, the pixel points in the diagonal direction of a can be selected to be added to the surrounding pixel points. If the first pixel point is b, when taking the point at the position with a difference of two pixels, it may exceed the image area, then the data of the nearest pixel point can be reused. For example, for point b, pixel points 5, 6, 7 and 8' can be taken as the surrounding pixel points, among which pixel point 8' reuses the data of pixel point 8. Figure 8 Only some pixels are shown as an example. In reality, the number of pixels in an image may be much larger than Figure 8 Quantity shown.

[0221] S703: Determine a target number of surrounding pixel points among N surrounding pixel points, the absolute value of the brightness difference between the brightness value of the first pixel point and the brightness value of the first pixel point being greater than a preset brightness threshold.

[0222] Specifically, the GPU uses the method in the above step S701 for the obtained N surrounding pixels to calculate the brightness value of each surrounding pixel, and then calculates the absolute value of the brightness difference between each surrounding pixel and the brightness value of the first pixel. Assuming that the coordinates of the first pixel are (u, v), the absolute value of the difference between Intensity (u, v) and Intensity (u-2, v), Intensity (u+2, v), Intensity (u, v-2), and Intensity (u, v+2) is calculated. Among them, Intensity represents the brightness value.

[0223] Taking the first pixel a as an example, determine the brightness values ​​of N surrounding pixels to obtain the brightness values ​​of pixels 1, 2, 3 and 4, and then calculate the brightness difference between the brightness values ​​of pixels 1, 2, 3 and 4 and the brightness value of the first pixel, that is, subtract the brightness value of the first pixel from the brightness values ​​of pixels 1, 2, 3 and 4, and obtain the brightness difference corresponding to each surrounding pixel, and then take the absolute value of each brightness difference, which represents the brightness difference between the corresponding surrounding pixel and the first pixel. Then, count the number of surrounding pixels whose absolute value is greater than the preset brightness threshold as the target number.

[0224] S704A: If the number of targets is greater than half of N, determine the first pixel point as a corner point.

[0225] S704B: If the number of targets is less than or equal to half of N, determine that the first pixel point is not a corner point.

[0226] If the number of targets is greater than half of N, it means that among the surrounding pixels around the first pixel, the number of points with a large brightness difference from the first pixel is in the majority, and the possibility that the first pixel is the focus is high, and the GPU determines that the first pixel is a corner point; if the number of targets is less than or equal to half of N, it means that among the surrounding pixels around the first pixel, the number of points with a large brightness difference from the first pixel is not in the majority, and the possibility that the first pixel is the focus is low, and the GPU determines that the first pixel is a corner point.

[0227] By adopting this method, each pixel point is screened according to the screening method of the first pixel point, and multiple pixel points that are not corner points can be deleted, leaving only the pixel points that are corner points.

[0228] Above Figure 7 In the method shown, the GPU can obtain the brightness value of each pixel according to the color image, and then screen out the corner points according to the difference between the brightness value of the pixel and the brightness values ​​of the surrounding pixels. A large number of non-corner pixels can be deleted, reducing the subsequent amount of calculations and saving power consumption.

[0229] In the above Figure 7 Based on the embodiment shown and related embodiments, after the corner points are obtained, the response function of each corner point can be calculated as follows: Fig. 9 As shown, including:

[0230] S901. Obtain a brightness difference of a first corner point, where the first corner point is any one of at least one corner point.

[0231] Taking the first corner point as an example, the two-dimensional image coordinates are labeled (u, v), and the GPU can obtain the brightness difference of the first corner point.

[0232] Optionally, the brightness difference may include a brightness difference in a first direction, or include a brightness difference in a second direction, or include a brightness difference in the first direction and a brightness difference in the second direction. The first direction is, for example, the x direction, the second direction may be the y direction, and the brightness difference Ix in the first direction may be the brightness value of the pixel point adjacent to the first corner point in the positive direction of the x direction minus the brightness value of the pixel point adjacent to the first corner point in the negative direction of the x direction, which may be expressed as follows:

[0233] Ix=Intensity0(u+1,v)-Intensity0(u-1,v);

[0234] The second direction brightness difference Iy can be the brightness value of the pixel points adjacent to the first corner point in the positive direction of the y direction minus the brightness value of the pixel points adjacent to the first corner point in the negative direction of the y direction, which can be expressed by the formula:

[0235] Iy=Intensity0(u,v+1)-Intensity0(u,v-1).

[0236] S902: Obtain a depth difference of a first corner point.

[0237] Taking the first corner point as an example, the two-dimensional image coordinates are marked as (u, v), and the GPU can obtain the depth difference of the first corner point.

[0238] Optionally, the depth difference may include a first-direction depth difference, or a second-direction depth difference, or a first-direction depth difference and a second-direction depth difference. The first direction is, for example, the x-direction, the second direction may be the y-direction, and the first-direction depth difference may be the depth value of the pixel point adjacent to the first corner point in the positive direction of the x-direction minus the depth value of the pixel point adjacent to the first corner point in the negative direction of the x-direction, which may be expressed as follows:

[0239] dx=Depth0(u+1,v)-Depth0(u-1,v);

[0240] The depth difference in the second direction can be the depth value of the pixel point adjacent to the first corner point in the positive direction of the y direction minus the depth value of the pixel point adjacent to the first corner point in the negative direction of the y direction, which can be expressed by the formula:

[0241] dy=Depth0(u,v+1)-Depth0(u,v-1).

[0242] Optionally, the depth difference and brightness difference are calculated with data at intervals of one pixel to the left or right or up or down, or with intervals of two pixels to the left or right or up or down, which is not limited in this embodiment of the present application.

[0243] It should be noted that the above depth values ​​are all taken from the depth image.

[0244] S903: Determine a response function corresponding to the first corner point according to the depth difference and the brightness difference.

[0245] Specifically, the GPU may determine a response function corresponding to the first corner point according to the depth difference and the brightness difference.

[0246] Optionally, the response function response may refer to the following formula or a variation of the formula:

[0247] response = exp(Ix, Iy, dx, dy); or responses = e lx,ly,dx,dy ;

[0248] This formula indicates that the response function is an expression containing an exponential function with the natural constant e as the base and the first exponent as the exponent, or is equal to an exponential function with the natural constant e as the base and the first exponent as the exponent. The first exponent is a function containing brightness difference and depth difference, for example, the first exponent is a function containing Ix, Iy, dx, and dy. e is the Euler number, and its value is approximately 2.718.

[0249] Specifically, the GPU performs the above operation on each corner point to obtain the value of the response function of each corner point.

[0250] Above Fig. 9 In the illustrated embodiment, the GPU uses a function of brightness difference and depth difference as the first exponent of e, and uses the first exponent power of e as the response function of the corresponding corner point. The obtained response function is strongly correlated with the brightness difference and the depth difference. Therefore, the value of the response function can accurately reflect the characteristics of the corner point.

[0251] After that, the GPU then calculates the value of the response function of each corner point, and then can filter out the corner points whose response function values ​​are greater than or equal to the preset response function threshold from the obtained corner points as the first feature points, and discard the corner points whose response function values ​​are less than the preset response function threshold. Thus, one or more first feature points can be obtained. Optionally, the preset response function threshold can be selected as 15.0, 14.5, etc. The specific value can be adjusted according to the function form of the first exponent, which is not limited here.

[0252] The GPU may also refer to the aforementioned embodiment and perform the feature point extraction operation on the subsequent frame image in the same manner as the previous frame image to obtain one or more second feature points, which will not be described in detail here.

[0253] Optionally, the GPU may also determine the number of matched feature point pairs. If the number of feature point pairs is too small, a small number of feature point pairs may not be able to fully express the overall motion trend of the global image. The GPU may only execute the subsequent process of determining the spatial transformation matrix based on the feature point pairs when the number of feature point pairs is greater than or equal to the preset feature point pair number threshold. If the number of feature point pairs is less than the preset feature point pair number threshold, the process is stopped and no longer processed; or the first feature point and the second feature point are re-identified, or the feature point matching is performed again until a sufficient number of feature point pairs are obtained, and the subsequent process is continued. Optionally, the preset feature point pair number threshold may be a natural number such as 6, 8 or 12, which is not limited in the embodiment of the present application. It should be noted that the larger the preset feature point pair number threshold, the more feature point pairs there are, the more accurate the motion vector obtained, the smaller the preset feature point pair number threshold, the fewer feature point pairs obtained, the less computation is required, and the more power consumption is saved. Specifically, the value can be balanced between accuracy and power consumption.

[0254] Based on the above embodiment, a possible implementation method of step S503 in the above embodiment may also include: determining a spatial transformation matrix based on at least one first feature point and at least one second feature point, the spatial transformation matrix representing the change in the spatial position of the same pixel point of the subsequent frame image relative to the previous frame image; performing singular value decomposition on the spatial transformation matrix to obtain rotation parameters and translation parameters.

[0255] Specifically, the GPU can perform feature point matching on at least one first feature point and at least one second feature point to obtain at least one feature point pair, and then generate a spatial transformation matrix based on the transformation relationship between the spatial positions of the two matched feature points in each feature point pair. Afterwards, the GPU also performs singular value decomposition on the spatial transformation matrix to separate rotation parameters and translation parameters. The at least one first feature point can be expressed by a feature point sequence, denoted as Keypoint1, and the at least one second feature point can be expressed by a feature point sequence, denoted as Keypoint2. The eigenvector of the feature point can be calculated based on Keypoint1 and Keypoint2. For details, please refer to the operation of eigenvector conversion below.

[0256] Optionally, the GPU may perform feature point matching on at least one first feature point and at least one second feature point, and may obtain multiple feature point pairs or one feature point pair. If multiple feature point pairs are obtained, a spatial transformation sub-matrix may be calculated for each feature point pair, and then the spatial transformation sub-matrices of the multiple feature point pairs may be combined into an overall spatial transformation matrix for subsequent singular value decomposition.

[0257] Optionally, the GPU performs feature point matching on at least one first feature point and at least one second feature point to obtain at least one set of feature point pairs. For example, one of the sets of feature point pairs is a first feature point pair, and the first feature point pair includes a first target feature point and a second target feature point. The first target feature point is any one of the set consisting of the at least one first feature point, and the second target feature point is a second feature point in the set consisting of the at least one second feature point that matches the first target feature point.

[0258] First, we will introduce how to perform feature point matching. Fig.10 As shown, including:

[0259] S1001. Perform a feature vector conversion operation on at least one first feature point to obtain at least one first feature vector, where the at least one first feature vector corresponds to the at least one first feature point in one-to-one correspondence.

[0260] Specifically, the GPU can perform a feature vector conversion operation on each of the at least one first feature point to obtain a first feature vector corresponding one-to-one to the at least one first feature point. The first feature vector is one or more, and the specific number is the same as the number of first feature points.

[0261] S1002: Perform a feature vector conversion operation on at least one second feature point to obtain at least one second feature vector, wherein the at least one second feature vector corresponds to the at least one second feature point in a one-to-one manner.

[0262] Specifically, the GPU can perform a feature vector conversion operation on each of the at least one second feature point to obtain a second feature vector corresponding one-to-one to the at least one second feature point, and the second feature vector is one or more, and the specific number is the same as the number of second feature points.

[0263] Optionally, the specific implementation of the above feature vector conversion operation can refer to the following description.

[0264] In order to support the feature vector conversion operation on the GPU, we first introduce the capabilities that the GPU needs to have. The compute shader on the GPU is used to calculate the feature vector, that is, to perform the feature vector conversion operation. SSBO0 (Shader Storage BufferObject) is a cache for all computing units provided to the compute shader. Usually, the GPU includes multiple computing cores, which can operate simultaneously, using the GPU's multi-core parallel computing capabilities to support multi-threaded work groups. Specifically, a 16*16 local work group can be constructed on the GPU, that is, a 256-size thread work group, corresponding to 256 threads. For details, please refer to Fig.11 As shown. This 16*16 local work group is a global work group of size (1,1,1). These 256 threads can share memory to implement the atomic operation atomicOr. Atomic operations are used between threads in the thread work group, which means that the running code logic is the same and can share the same set of code. When multiple threads of the compute shader need to read and write data, due to shared memory, the data in the memory area will be unlocked or locked during the atomic operation to prevent multiple threads from reading and writing the same memory location at the same time and causing confusion.

[0265] Specifically, for each feature point, a point in a window of a preset size can be taken to calculate the feature vector. The size of the window can be the default size of 20*20. Optionally, the size of the window can also change with the size of the image itself, which can be determined according to the incoming scaling factor. For example, if the image has a pixel size of 1280*720, the scale can be 1, and the window size is 20*20; if the image has a pixel size of 512*512, the scale can be 0.5, and the window size is 10*10. The size of the window can be adjusted according to the incoming scaling factor scale to ensure that the value is reasonable. The scaling factor can also ensure that the window does not exceed the size of the original image.

[0266] Based on the above GPU capabilities, the specific logic of the feature vector conversion operation is as follows: the GPU creates 256 four-dimensional vectors, each of which requires a thread to implement the operation. Each thread is marked by the corresponding subscript gl_LocalInvocationIndex, for example, gl_LocalInvocation1 represents Fig.11 The thread corresponding to the first position in the upper left corner of the gl_LocalInvocation2 represents Fig.11 The second position of the first line in the table corresponds to the thread. These 256 threads can be marked with gl_LocalInvocationIndex.

[0267] Each thread can calculate a random vector based on its own subscript. Specifically, it can be to multiply the subscript of its own thread by a fixed value, then take the trigonometric function value as the exponent, and the natural constant e as the base to get an exponential function, and take the value of the exponential function as an integer as a number of the four-dimensional vector. When four different exponential functions are used for calculation, four integers can be obtained, and these four numbers can be used as random numbers of a four-dimensional vector. Examples of four different exponential functions are: e sin(gl_localInvocationID*73456) , e cos(gl_localInvocationID*73456) , e tan(gl _localInvocationID*73456) , e cot(gl_localInvocationID*73456) The fixed value is 73456. Optionally, the fixed value can also be other values. Fig.11 When the thread corresponds to the first position in the upper left corner, gl_LocalInvocationID takes 1. Fig.11 If it is the last position in the lower right corner, the ID is 256. Fig.11 The thread subscript in is also an example and is not used to limit the form of the subscript. Fig.11 The numbers in each square matrix can be the subscripts of the corresponding threads. For example, the subscripts of each thread are natural numbers in sequence, so the subscripts corresponding to 256 threads are natural numbers from 1 to 256. Fig.11 Some subscripts are marked as examples, and the rest are not marked one by one. Using four exponential functions, a corresponding four-dimensional vector can be calculated for each thread group, which is called a random vector and can be expressed as [a, b, c, d]. In this way, each thread can calculate a four-dimensional random vector based on its own subscript, so 256 threads can calculate 256 random vectors.

[0268] Next, these 256 random vectors are used to perform feature vector conversion on the feature point. Take a feature point with coordinates (u, v) in a two-dimensional image as an example. The calculation starts from the first random vector among the above 256 random vectors, that is, the random vector corresponding to gl_LocalInvocation1. The random vector corresponding to gl_LocalInvocation1 can be recorded as [a1, b1, c1, d1]. Then, the offset coordinates [a1, b1] of offset point 1 and the offset coordinates [c1, d1] of offset point 2 are determined according to the random vector [a1, b1, c1, d1]. Then, the brightness value of the pixel point with coordinates (u-a1, v-b1) is calculated, which is recorded as brightness value 1. Among them, the pixel point with coordinates (u-a1, v-b1) represents the coordinates of the feature point coordinates (u, v) offset [a1, b1]. The brightness value of the pixel point with coordinates (u-c1, v-d1) is also calculated, which is recorded as brightness value 2. Among them, the pixel point with coordinates (u-c1, v-d1) represents the coordinates of the feature point coordinates (u, v) offset [c1, d1]. Then compare the brightness value 1 and the brightness value 2. If the brightness value 1 is greater than the brightness value 2, the current output value is determined to be 1; if the brightness value 1 is less than the brightness value 2, the current output value is determined to be 0. The output value is the number in the feature vector converted from the current random vector [a1, b1, c1, d1].

[0269] It should be noted that the offset obtained by the above random vector can be taken within a window centered on the feature point (u, v).

[0270] The brightness value 1 and the brightness value 2 can be obtained by first obtaining the color value of the corresponding position, then calculating the brightness value according to the color value, and finally using atomic operations to calculate the magnitude relationship between the brightness value 1 and the brightness value 2 to obtain the number of the gl_LocalInvocationIndex position of the feature vector. This can be done by the following formula or a variation of the formula:

[0271] Color0_i.rgba=texture(Color0,gl_GlobalInvocationID.xy+offsetUV_i);

[0272] Among them, Color0_i.rgba means: in the color image Color0, the feature point with coordinates (x, y) corresponds to the color value of the point with coordinate offset offsetUV_i corresponding to the thread with subscript GlobalInvocationi. texture is used to mark the image ID as Color0. Based on the above color value, the corresponding brightness value can be calculated, referring to the following formula:

[0273] Intensity_i=Color0_i.r*0.299+Color0_i*0.587+Color0_i*0.114.

[0274] If the random vector [a1, b1, c1, d1] is a random vector generated by the first thread, the output value corresponds to the first value of the feature vector in the feature vector. If the random vector is a random vector generated by the second thread, the output value corresponds to the second value of the feature vector in the feature vector. If the random vector is a random vector generated by the Nth thread, the output value corresponds to the Nth value of the feature vector in the feature vector.

[0275] After all threads have completed the calculation, a feature vector specialVector with a length of 256 and composed of 0s and 1s can be generated. The expression statement is: Atomic(specialVector[gl_LocalInvocationIndex], result) to obtain the feature vector corresponding to the feature point, that is, to complete a feature vector conversion operation.

[0276] Optionally, in the process of calculating the feature vector, the feature vector may be generated based on the brightness value combined with the depth value of the corresponding pixel point to improve the accuracy of the feature vector.

[0277] S1003. Determine, in at least one second eigenvector, that the second eigenvector with the smallest spatial distance to the first target eigenvector is the second target eigenvector, the first target eigenvector is the first eigenvector corresponding to the first target feature point, the second target eigenvector is the second eigenvector corresponding to the second target feature point, and the feature point pair formed by the first target feature point and the second target feature point is a pair in at least one set of feature point pairs.

[0278] Optionally, the GPU performs feature point matching on at least one first feature point and at least one second feature point to obtain at least one set of feature point pairs. For example, one of the sets of feature point pairs is a first feature point pair, and the first feature point pair includes a first target feature point and a second target feature point. The first target feature point is any one of the set consisting of the at least one first feature point, and the second target feature point is a second feature point in the set consisting of the at least one second feature point that matches the first target feature point.

[0279] The GPU selects a first target feature vector from at least one first feature vector, and then calculates the vector distance between each second feature vector and the first target feature vector, and then selects the second feature vector corresponding to the smallest vector distance, recorded as the second target feature vector, and then determines that the second target feature point corresponding to the second target feature vector and the first target feature point are two matching feature points.

[0280] The GPU needs to perform the same operation as the first target feature vector for each first feature vector, and obtain the second feature point matched by the first feature point corresponding to each first feature vector, thereby obtaining one or more sets of feature point pairs.

[0281] Using the above Fig.10 In the method shown, the GPU converts feature points into feature vectors and then matches the feature points according to the feature distance between the feature vectors. The matching degree is high, so that the processing of subsequent steps is more accurate.

[0282] Secondly, after matching the at least one first feature point and the at least one second feature point to obtain at least one set of feature point pairs, a spatial transformation matrix is ​​determined based on the at least one set of feature point pairs. For details, see Fig.12 The process shown here illustrates how to obtain the space transformation matrix, including:

[0283] S1201. Obtain first centroid coordinates of at least one first feature point.

[0284] After the GPU identifies the first feature point of the previous frame image, if the number of the first feature point is one, the coordinates of the first feature point are the first centroid coordinates. If the number of the first feature points is multiple, the centroid coordinates of the multiple first feature points can be obtained and recorded as the first centroid coordinates. For example, it can be obtained according to the following formula or a variation of the formula:

[0285] viewPosC.xyz=∑(viewPos.xyz) / keyPoint0.Num;

[0286] Among them, viewPosC.xyz represents the first centroid coordinates, viewPos.xyz is the 3D world coordinates of each first feature point, and keyPoint0.Num represents the number of first feature points. The meaning of this formula is to sum the 3D world coordinates of each first feature point and divide it by the number of first feature points to get the first centroid coordinates.

[0287] S1202: Obtain a first centroid offset of a first target feature point relative to a first centroid coordinate, where the first target feature point is any one of the at least one first feature point.

[0288] A first target feature point is selected from the plurality of first feature points to determine a first centroid offset of the first target feature point relative to the first centroid coordinates. The centroid offset can also be called the de-centroid coordinates. For example, it can be obtained according to the following formula or a variation of the formula:

[0289] viewPos.xyz(centroid) = viewPos.xyz - viewPosC.xyz;

[0290] Among them, viewPos.xyz (centroid removed) represents the first centroid offset, viewPos.xyz represents the three-dimensional world coordinates of the first target feature point, and viewPosC.xyz represents the first centroid coordinates.

[0291] The GPU can calculate the corresponding first centroid coordinates for each first feature point, that is, a corresponding first centroid offset can be calculated for each first feature point, and the first centroid offset represents the offset of the corresponding first feature point relative to the first centroid coordinates.

[0292] S1203. Obtain second centroid coordinates of at least one second feature point.

[0293] After the GPU identifies the second feature point of the next frame image, if the number of the second feature point is one, the coordinates of the second feature point are the second centroid coordinates. If the number of the second feature point is multiple, the centroid coordinates of the multiple second feature points can be obtained and recorded as the second centroid coordinates.

[0294] S1204. Obtain a second center of mass offset of a second target feature point relative to the second center of mass coordinates, where the second target feature point is a second feature point among at least one second feature point that matches the first target feature point, and a feature point pair formed by the first target feature point and the second target feature point is any pair in at least one set of feature point pairs.

[0295] A second feature point matching the first target feature point is selected from the plurality of second feature points, recorded as the second target feature point, to determine the second centroid offset of the second target feature point relative to the second centroid coordinates, the centroid offset can also be called the centroid coordinate. For details, refer to the formula or the variation of the formula in step S1002, and the coordinates of the corresponding feature point in the formula need to be replaced with the coordinates of the feature point in the next image, which will not be repeated here.

[0296] The GPU can calculate the corresponding second center of mass coordinates for each second feature point, that is, a corresponding second center of mass offset can be calculated for each second feature point, and the second center of mass offset represents the offset of the corresponding second feature point relative to the first center of mass coordinates.

[0297] S1205: Multiply each of the multiple directional offset values ​​in the first centroid offset by each of the multiple directional offset values ​​in the second centroid offset in pairs to obtain a spatial transformation matrix.

[0298] Specifically, the GPU calculates, for the first target feature point and the second target feature point as a set of feature point pairs, each of the offset values ​​in each direction in the first center of mass offset corresponding to the first target feature point, and multiplies each of the offset values ​​in each direction in the second center of mass offset corresponding to the second target feature point by two-by-two, to obtain the spatial transformation sub-matrix corresponding to the set of feature point pairs.

[0299] Specifically, the first centroid offset corresponding to the first target feature point is expressed as viewPos0.xyz (excluding the centroid), and the offset values ​​of the three directions in viewPos0.xyz (excluding the centroid) are viewPos0.x, viewPos0.y, and viewPos0.z. The second centroid offset corresponding to the second target feature point is expressed as viewPos1.xyz (excluding the centroid), and the offset values ​​of the three directions in viewPos1.xyz (excluding the centroid) are viewPos1.x, viewPos1.y, and viewPos1.z. Then, the GPU can multiply viewPos0.x, viewPos0.y, viewPos0.z and viewPos1.x, viewPos1.y, viewPos1.z by two by two, that is, bring them into the following matrix expression:

[0300]

[0301] Among them, Martix represents the spatial transformation submatrix corresponding to the first feature point pair composed of the first target feature point and the second target feature point. Optionally, if only the first feature point pair exists, the spatial transformation submatrix is ​​the above-mentioned spatial transformation matrix; if feature point matching is performed on at least one first feature point and at least one second feature point to obtain multiple groups of feature point pairs, the other feature point pairs can also refer to the first feature point pair to obtain the spatial transformation submatrix corresponding to each feature point pair, and then combine the spatial transformation submatrices obtained for each feature point pair into a spatial transformation matrix, and the specific combination form may not be limited. Optionally, the spatial transformation submatrices can be added to obtain the overall spatial transformation matrix.

[0302] Above Fig.12 In the embodiment shown, the GPU can collect the offset of each feature point from the centroid coordinates in the next frame of the image based on the offset of each feature point from the centroid coordinates in the previous frame of the image, and can fully combine the offset changes of the matched feature points in various directions, so that the obtained spatial transformation matrix can accurately reflect the offset trend of the feature points in space. Therefore, compared with the spatial transformation matrix generated by matching all pixel points to obtain each pair of matched pixel points, the number of processed pixel points is greatly reduced, the amount of calculation is reduced, and power consumption is saved.

[0303] In order to fully describe the technical solution of this application, Fig.13 , a unified description of the vector determination method of the embodiment of the present application is given, and the details can be found in Fig.13 ,include:

[0304] S1301. Obtain a Hook instruction stream of a rendering resource to obtain a color image and a depth image.

[0305] The Hook instruction stream is also the rendering quality stream, and the GPU can extract the color image and the depth image from the Hook instruction stream. Here, the GPU extracts the respective color image and the respective depth image for the Hook instruction stream corresponding to the previous frame image and the next frame image respectively.

[0306] S1302: Perform feature point recognition on the color image, and obtain feature points by combining the depth image and the projection matrix.

[0307] Specifically, the GPU can perform feature point recognition on the two color images of the previous frame image and the next frame image respectively to obtain the feature points of each frame image. For example, a feature point extraction operation can be performed on the color image to obtain at least one first feature point of the previous frame image and at least one second feature point of the next frame image. The specific method of the feature point extraction operation can be referred to as described above, and will not be repeated here. It should be noted that the coordinates of the obtained feature points can be expressed as three-dimensional space coordinates, such as three-dimensional camera coordinates expressed in a camera coordinate system.

[0308] S1303: Determine whether the number of feature points is greater than or equal to a preset feature point number threshold. If yes, execute S1304; if no, stop the process.

[0309] Specifically, the GPU may determine whether the number of first feature points corresponding to the previous frame image is greater than or equal to a preset feature point number threshold, and may also determine whether the number of second feature points corresponding to the next frame image is greater than or equal to a preset feature point number threshold. When both the number of the first feature points and the number of the second feature points are greater than or equal to the preset feature point number threshold, it is determined to execute S1304.

[0310] S1304: Perform feature point matching to obtain feature point pairs.

[0311] Specifically, the GPU performs feature point matching on the first feature point obtained from the previous frame image and the second feature point obtained from the next frame image. The specific method of performing feature point matching can be referred to above and will not be described here. The obtained feature point pairs may be one or more groups.

[0312] S1305, determine whether the number of feature point pairs is greater than a preset feature point pair number threshold. If yes, execute S1306, if not, stop the process.

[0313] S1306: Construct a spatial transformation matrix according to the feature point pairs.

[0314] Specifically, the GPU may construct a spatial transformation matrix according to the obtained multiple groups of feature point pairs. The specific process may also refer to the description of the above embodiment, which will not be described here.

[0315] S1307. Perform singular value decomposition on the space transformation matrix to obtain rotation parameters and translation parameters.

[0316] S1308. Reproject the coordinates of the feature points to obtain predicted two-dimensional image coordinates.

[0317] Specifically, the GPU performs motion interpolation on the rotation and translation parameters, that is, multiplying by 0.5 as mentioned above, as the rotation and translation amount of the pixel in the predicted frame. Among them, the coordinate reprojection is that the GPU transforms the three-dimensional camera coordinates of a feature point of one color image from three-dimensional to two-dimensional coordinates according to half of the rotation parameter and half of the translation parameter, and obtains the predicted two-dimensional image coordinates of the feature point in the predicted frame. The specific process can also be found in the description of the previous embodiment, which will not be repeated here.

[0318] S1309: Obtain a motion vector according to the predicted two-dimensional image coordinates.

[0319] Specifically, the GPU may determine the difference between the predicted two-dimensional image coordinates of the feature point and the two-dimensional image coordinates of the feature point in the previous frame image as a motion vector.

[0320] Fig.13 The explanation of the specific steps of the method shown, the implementation principle and the beneficial effects can all be referred to the above embodiments, which will not be repeated here. The method obtains a small amount of information from the rendering instruction stream, such as the projection matrix, color image and depth image of the virtual camera, and can efficiently calculate and generate motion vectors with high accuracy. Taking game applications as an example, the data dependence on the game application is low, which can not only improve efficiency but also save power consumption.

[0321] The above describes in detail an example of the method provided by the present application. It is understandable that, in order to implement the above functions, the corresponding device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0322] The present application can divide the functional modules of the motion vector determination device according to the above method example. For example, each function can be divided into each functional module, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0323] Fig.14 The schematic diagram of the structure of a motion vector determination device provided by the present application is shown. The device 1400 includes:

[0324] The extraction module 1401 is used to perform a feature point extraction operation on a previous frame image to obtain at least one first feature point, and to perform a feature point extraction operation on a next frame image to obtain at least one second feature point.

[0325] The generating module 1402 is used to determine the rotation parameters and translation parameters of the virtual camera according to the at least one first feature point and the at least one second feature point, and the virtual camera is used to obtain the previous frame image and the next frame image.

[0326] The determination module 1403 is used to determine a motion vector according to the rotation parameter and the translation parameter, where the motion vector represents the motion trend of the pixel point.

[0327] In some embodiments, the extraction module 1401 is specifically used to obtain at least one corner point in the target image, where the target image is the previous frame image or the next frame image, and the at least one corner point is a pixel point with large brightness change in the target image; and determine that among the at least one corner point, the corner point whose response function value is greater than or equal to a preset response function threshold is a target feature point, and the target feature point is any one of the at least one first feature point or any one of the at least one second feature point.

[0328] In some embodiments, the extraction module 1401 is specifically used to obtain a color image of the target image; and perform a corner point selection operation on each pixel point in the color image to obtain at least one corner point.

[0329] In some embodiments, a color image carries color values ​​of multiple pixels, and an extraction module 1401 is specifically used to determine a brightness value of the first pixel based on the color value of the first pixel, where the first pixel is any one of the multiple pixels in the color image; determine the brightness values ​​of N surrounding pixels based on the color values ​​of N surrounding pixels, where the N surrounding pixels are pixels distributed around the first pixel, and N is a natural number; determine a target number of surrounding pixels among the N surrounding pixels, where the absolute value of the brightness difference between the brightness value and the brightness value of the first pixel is greater than a preset brightness threshold; if the target number is greater than half of N, determine that the first pixel is a corner point; if the target number is less than or equal to half of N, determine that the first pixel is not a corner point.

[0330] In some embodiments, the extraction module 1401 is also used to obtain the brightness difference of the first corner point before determining that the corner point whose response function value is greater than a preset response function threshold is a target feature point in at least one corner point; the first corner point is any one of the at least one corner point; obtain the depth difference of the first corner point; and determine the response function corresponding to the first corner point based on the depth difference and the brightness difference.

[0331] In some embodiments, the brightness difference includes a first-direction brightness difference and a second-direction brightness difference, the first-direction brightness difference being the difference in brightness values ​​of two pixels on both sides of a first corner point in the first direction, and the second-direction brightness difference being the difference in brightness values ​​of two pixels on both sides of the first corner point in the second direction; the depth difference includes a first-direction depth difference and a second-direction depth difference, the first-direction depth difference being the difference in depth values ​​of two pixels on both sides of the first corner point in the first direction, and the second-direction depth difference being the difference in depth values ​​of two pixels on both sides of the first corner point in the second direction.

[0332] In some embodiments, the response function corresponding to the first corner point includes an exponential function with a natural constant e as a base and a first exponent as an exponent, and the first exponent is a function including a brightness difference and a depth difference.

[0333] In some embodiments, the generation module 1402 is used to determine a spatial transformation matrix based on at least one first feature point and at least one second feature point, where the spatial transformation matrix represents the change in the spatial position of the same pixel point of the subsequent frame image relative to the previous frame image; and perform singular value decomposition on the spatial transformation matrix to obtain rotation parameters and translation parameters.

[0334] In some embodiments, the generation module 1402 is specifically used to perform feature point matching on at least one first feature point and at least one second feature point to obtain at least one set of feature point pairs, any one set of the first feature point pairs in at least one set of feature point pairs, the first feature point pairs include a first target feature point and a second target feature point, the first target feature point is any one of the at least one first feature point, and the second target feature point is a second feature point in the at least one second feature point that matches the first target feature point; based on at least one set of feature point pairs, determine the spatial transformation matrix.

[0335] In some embodiments, the generation module 1402 is specifically used to perform a feature vector conversion operation on at least one first feature point to obtain at least one first feature vector, and the at least one first feature vector corresponds to the at least one first feature point in one-to-one correspondence; perform a feature vector conversion operation on at least one second feature point to obtain at least one second feature vector, and the at least one second feature vector corresponds to the at least one second feature point in one-to-one correspondence; determine that the second feature vector with the smallest spatial distance to the first target feature vector in the at least one second feature vector is the second target feature vector, the first target feature vector is the first feature vector corresponding to the first target feature point, the second target feature vector is the second feature vector corresponding to the second target feature point, and the feature point pair formed by the first target feature point and the second target feature point is a pair in at least one set of feature point pairs.

[0336] In some embodiments, the generation module 1402 is specifically used to obtain multiple random vectors; extract the first offset point and the second offset point according to the first random vector, the first random vector is any one of the multiple random vectors; obtain the brightness value of the first offset point and the brightness value of the second offset point; output the first vector parameter according to the size relationship between the brightness value of the first offset point and the brightness value of the second offset point; when the brightness value of the first offset point is greater than the brightness value of the second offset point, the first vector parameter is 1; when the brightness value of the first offset point is less than the brightness value of the second offset point, the first vector parameter is 0; the first vector parameter is a parameter in the first eigenvector, the first eigenvector also includes other vector parameters, the other vector parameters are vector parameters determined according to other random vectors, and the other random vectors are random vectors among the multiple random vectors.

[0337] In some embodiments, generation module 1402 is specifically used to obtain the first center of mass coordinates of at least one first feature point; obtain the first center of mass offset of the first target feature point relative to the first center of mass coordinates, the first target feature point is any one of the at least one first feature point; obtain the second center of mass coordinates of at least one second feature point; obtain the second center of mass offset of the second target feature point relative to the second center of mass coordinates, the second target feature point is a second feature point in at least one second feature point that matches the first target feature point, and the feature point pair formed by the first target feature point and the second target feature point is any pair in at least one set of feature point pairs; multiply each of the multiple directional offset values ​​in the first center of mass offset by each of the multiple directional offset values ​​in the second center of mass offset to obtain a spatial transformation matrix.

[0338] In some embodiments, the determination module 1403 is specifically used to obtain the two-dimensional image coordinates of a reference feature point, where the reference feature point is any one of at least one first feature point or at least one second feature point; determine the three-dimensional camera coordinates of the reference feature point in the camera space according to the depth image of the target image where the reference feature point is located and the projection matrix of the target image; predict the predicted three-dimensional camera coordinates of the reference feature point according to the three-dimensional camera coordinates of the reference feature point combined with the rotation parameters and the translation parameters; convert the predicted three-dimensional camera coordinates into predicted two-dimensional image coordinates according to the world matrix and the projection matrix of the virtual camera; determine the motion vector according to the two-dimensional image coordinates of the reference feature point and the predicted two-dimensional image coordinates.

[0339] In some embodiments, the reference feature point is any one of the at least one first feature point.

[0340] The specific manner in which the device 1400 executes the motion vector determination method and the beneficial effects produced can be found in the relevant description of the method embodiment, which will not be repeated here.

[0341] This embodiment further provides a chip, which includes a processor; the processor is used to read and execute a computer program stored in a memory to execute the methods of each of the above embodiments.

[0342] Optionally, the chip is a GPU.

[0343] The present application also provides an electronic device, including the above processor. The electronic device provided by this embodiment can be Figure 1The terminal device 100 shown is used to perform the above-mentioned motion vector determination method. In the case of an integrated unit, the terminal device may include a processing module, a storage module and a communication module. Among them, the processing module can be used to control and manage the actions of the terminal device, for example, it can be used to support the terminal device to execute the steps performed by the display unit, the detection unit and the processing unit. The storage module can be used to support the terminal device to execute the stored program code and data, etc. The communication module can be used to support the communication between the terminal device and other devices.

[0344] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processor (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other terminal devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0345] In one embodiment, when the processing module is a processor and the storage module is a memory, the terminal device involved in this embodiment may be a Figure 1 Device of the structure shown.

[0346] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the motion vector determination method described in any of the above embodiments.

[0347] The embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the motion vector determination method in the above-mentioned embodiment.

[0348] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0349] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, the replaced units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0350] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0351] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0352] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining a motion vector, characterized in that: include: Performing a feature point extraction operation on the previous frame image to obtain at least one first feature point; Performing the feature point extraction operation on the next frame image to obtain at least one second feature point; Determine, according to the at least one first feature point and the at least one second feature point, a rotation parameter and a translation parameter of a virtual camera, wherein the virtual camera is used to acquire the previous frame image and the next frame image; A motion vector is determined according to the rotation parameter and the translation parameter, where the motion vector represents a motion trend of the pixel.

2. The method according to claim 1, characterized in that The feature point extraction operation includes: Acquire at least one corner point in a target image, wherein the target image is the previous frame image or the next frame image, and the at least one corner point is a pixel point with a large brightness change in the target image; Determine, among the at least one corner point, a corner point whose response function value is greater than or equal to a preset response function threshold as a target feature point, wherein the target feature point is any one of the at least one first feature point or any one of the at least one second feature point.

3. The method according to claim 2, characterized in that The obtaining of at least one corner point in the target image comprises: Acquire a color image of the target image; A corner point selection operation is performed on each pixel point in the color image to obtain the at least one corner point.

4. The method according to claim 3, characterized in that The color image carries color values ​​of multiple pixels, and the corner point selection operation includes: Determining a brightness value of a first pixel according to a color value of the first pixel, where the first pixel is any one of the plurality of pixels in the color image; Determine brightness values ​​of N surrounding pixels according to color values ​​of the N surrounding pixels, where the N surrounding pixels are pixels distributed around the first pixel, and N is a natural number; Determine a target number of the N surrounding pixel points whose absolute value of the brightness difference between the brightness value of the first pixel point and the brightness value of the first pixel point is greater than a preset brightness threshold; If the number of targets is greater than half of N, determining the first pixel point as the corner point; If the target number is less than or equal to half of N, it is determined that the first pixel point is not the corner point.

5. The method according to claim 2, characterized in that: Before determining, among the at least one corner point, that a corner point whose response function value is greater than a preset response function threshold is a target feature point, the method includes: Obtaining a brightness difference of a first corner point, where the first corner point is any one of the at least one corner point; Obtaining a depth difference of the first corner point; A response function corresponding to the first corner point is determined according to the depth difference and the brightness difference.

6. The method according to claim 5, characterized in that The brightness difference includes a first direction brightness difference and a second direction brightness difference, wherein the first direction brightness difference is a difference between brightness values ​​of two pixels on both sides of the first corner point in the first direction, and the second direction brightness difference is a difference between brightness values ​​of two pixels on both sides of the first corner point in the second direction; The depth difference includes a first-direction depth difference and a second-direction depth difference. The first-direction depth difference is a difference in depth values ​​of two pixel points on both sides of the first corner point in the first direction. The second-direction depth difference is a difference in depth values ​​of two pixel points on both sides of the first corner point in the second direction.

7. The method according to claim 6, characterized in that The response function corresponding to the first corner point includes an exponential function with a natural constant e as a base and a first exponent as an exponent, and the first exponent is a function including the brightness difference and the depth difference.

8. The method according to any one of claims 1 to 7, characterized in that The determining, according to the at least one first feature point and the at least one second feature point, a rotation parameter and a translation parameter of the virtual camera comprises: Determine a spatial transformation matrix according to the at least one first feature point and the at least one second feature point, wherein the spatial transformation matrix represents a change in a spatial position of a same pixel point of the subsequent frame image relative to the previous frame image; Performing singular value decomposition on the spatial transformation matrix to obtain the rotation parameter and the translation parameter.

9. The method according to claim 8, characterized in that The determining of a spatial transformation matrix according to the at least one first feature point and the at least one second feature point comprises: Performing feature point matching on the at least one first feature point and the at least one second feature point to obtain at least one set of feature point pairs, wherein the first feature point matches any one of the at least one set of feature point pairs, wherein the first feature point pair includes a first target feature point and a second target feature point, wherein the first target feature point is any one of the at least one first feature point, and the second target feature point is a second feature point in the at least one second feature point that matches the first target feature point; The spatial transformation matrix is ​​determined according to the at least one set of feature point pairs.

10. The method according to claim 9, characterized in that The performing feature point matching on the at least one first feature point and the at least one second feature point to obtain at least one set of feature point pairs includes: Performing a feature vector conversion operation on the at least one first feature point to obtain at least one first feature vector, wherein the at least one first feature vector corresponds to the at least one first feature point in a one-to-one manner; Performing the feature vector conversion operation on the at least one second feature point to obtain at least one second feature vector, wherein the at least one second feature vector corresponds to the at least one second feature point in a one-to-one manner; Determine, among the at least one second eigenvector, the second eigenvector having the smallest spatial distance to the first target eigenvector as the second target eigenvector, the first target eigenvector being the first eigenvector corresponding to the first target feature point, the second target eigenvector being the second eigenvector corresponding to the second target feature point, and the feature point pair formed by the first target feature point and the second target feature point being a pair in the at least one set of feature point pairs.

11. The method according to claim 10, characterized in that The feature vector conversion operation includes: Get multiple random vectors; Extracting a first offset point and a second offset point according to a first random vector, wherein the first random vector is any one of the multiple random vectors; Obtaining a brightness value of the first offset point and a brightness value of the second offset point; Outputting a first vector parameter according to a magnitude relationship between a brightness value of the first offset point and a brightness value of the second offset point; When the brightness value of the first offset point is greater than the brightness value of the second offset point, the first vector parameter is 1; When the brightness value of the first offset point is less than the brightness value of the second offset point, the first vector parameter is 0; The first vector parameter is a parameter in the first feature vector. The first feature vector also includes other vector parameters. The other vector parameters are vector parameters determined according to other random vectors. The other random vectors are random vectors among the multiple random vectors.

12. The method according to any one of claims 9 to 11, characterized in that The determining the spatial transformation matrix according to the at least one set of feature point pairs comprises: Acquire the first centroid coordinates of the at least one first feature point; Acquire a first centroid offset of a first target feature point relative to the first centroid coordinate, where the first target feature point is any one of the at least one first feature point; Acquire the second centroid coordinates of the at least one second feature point; Acquire a second centroid offset of a second target feature point relative to the second centroid coordinate, where the second target feature point is a second feature point in the at least one second feature point that matches the first target feature point, and a feature point pair formed by the first target feature point and the second target feature point is any pair in the at least one set of feature point pairs; Each of the multiple direction offset values ​​in the first center of mass offset and each of the multiple direction offset values ​​in the second center of mass offset are multiplied in pairs to obtain the space transformation matrix.

13. The method according to any one of claims 1 to 12, characterized in that The determining of the motion vector according to the rotation parameter and the translation parameter comprises: Acquire the two-dimensional image coordinates of a reference feature point, where the reference feature point is any one of the at least one first feature point or the at least one second feature point; Determine the three-dimensional camera coordinates of the reference feature point in the camera space according to the depth image of the target image where the reference feature point is located and the projection matrix of the target image; Predicting predicted 3D camera coordinates of the reference feature point based on the 3D camera coordinates of the reference feature point in combination with the rotation parameter and the translation parameter; Converting the predicted three-dimensional camera coordinates into predicted two-dimensional image coordinates according to the world matrix of the virtual camera and the projection matrix; The motion vector is determined according to the two-dimensional image coordinates of the reference feature point and the predicted two-dimensional image coordinates.

14. The method according to claim 13, characterized in that The reference feature point is any one of the at least one first feature point.

15. A chip, characterized in that: The chip includes a processor; the processor is used to read and execute a computer program stored in a memory to perform the method according to any one of claims 1 to 14.

16. The chip according to claim 15, characterized in that: The chip is a graphics processing unit GPU.

17. An electronic device, characterized in that: include: Processors, memory, and interfaces; The processor, the memory and the interface cooperate with each other so that the electronic device executes the method as described in any one of claims 1 to 14; or includes the chip as described in claim 15 or 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 14.

Images