Image processing method and electronic equipment

Abstract

The invention discloses an image processing method and electronic equipment, relates to the technical field of image processing, and can improve the accuracy of predicting a frame image in an application scene of predicting an image through a motion compensation technology. The method comprises the steps that a first image is rendered, and the first image comprises a first pixel point and a second pixel point; and predicting a second image based on the first image, the second image including a third pixel point, and predicting that the positions of the first pixel point and the second pixel point after moving according to respective motion vectors correspond to the position of the third pixel point in the second image. And respectively displaying the first image and the second image, wherein information in the pixel point closest to the virtual camera in the first pixel point and the second pixel point is displayed in a third pixel point of the second image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0002] In recent years, in order to meet the high frame rate requirements of game applications, electronic devices can use a combination of image rendering and frame prediction technology to obtain more images per second. Among them, frame prediction technology refers to the technology of obtaining a predicted frame image based on two known frames of images.

[0003] A typical frame prediction technology is motion compensation technology. Using motion compensation technology, a predicted frame image can be obtained based on the motion relationship between pixels in two known frames of images.

[0004] However, in practice, multiple pixels in two known frames may move to the same position in the predicted frame. The existing motion compensation technology cannot solve the problem of selecting and determining when multiple pixels move to the same pixel. In other words, it is impossible to determine which pixel in the predicted frame actually displays the information of the same pixel. Summary of the invention

[0005] The present application provides an image processing method and an electronic device. In an application scenario where an image is predicted using motion compensation technology, even if multiple pixels move to the same pixel of a predicted frame image, information from a pixel closest to a camera among multiple pixels can be displayed at the same pixel, thereby improving the accuracy of the predicted frame image.

[0006] In a first aspect, the present application provides an image processing method. A first image is rendered, and the first image includes a first pixel and a second pixel. A second image is predicted based on the first image, and the second image includes a third pixel. It is predicted that the positions of the first pixel and the second pixel after being moved according to their respective motion vectors correspond to the position of the third pixel in the second image, and the motion vector is obtained based on at least two frames of images rendered, and the at least two frames of images include the first image. The first image and the second image are displayed respectively. If the first distance between the first pixel and the virtual camera when rendering the first image is less than the second distance between the second pixel and the virtual camera, the third pixel and the first pixel display the same information; if the first distance is greater than the second distance, the third pixel and the second pixel display the same information; the distance between the pixel and the virtual camera includes: the distance between the three-dimensional coordinates of the pixel in the camera coordinate system of the virtual camera and the virtual camera.

[0007] In summary, by adopting the present application, when the positions of multiple pixel points (such as the first pixel point and the second pixel point) in the rendered real frame image (such as the first image) after movement according to their respective motion vectors correspond to the same pixel point (such as the third pixel point) in the predicted frame image (such as the second image), the data of the pixel point closest to the virtual camera among the multiple pixel points can be written into the same pixel point. In this way, it is possible to avoid presenting information of pixel points farther from the virtual camera in the same pixel point of the predicted frame image, thereby ensuring the correctness of the predicted frame image.

[0008] In a possible design of the first aspect, the method is applied to an electronic device, wherein the electronic device includes a correspondence relationship between pixels in a first image and pixels in a second image, wherein a third pixel in the second image corresponds to a pixel closest to a virtual camera between the first pixel and the second pixel. Predicting the second image based on the first image includes: writing data in a pixel corresponding to the third pixel in the correspondence relationship to the third pixel.

[0009] That is to say, by recording the correspondence between the pixel point closest to the virtual camera among multiple pixel points and the above-mentioned same pixel point in the predicted frame image, the data of the pixel point corresponding to the above-mentioned same pixel point in the correspondence can be obtained and written into the above-mentioned same pixel point, and the data of the pixel point closest to the virtual camera can be accurately written into the above-mentioned same pixel point.

[0010] In a possible design of the first aspect, the electronic device uses a third image to represent the corresponding relationship, the pixels in the third image correspond one-to-one with the pixels in the second image, the fourth pixel of the third image corresponds to the third pixel, the fourth pixel records the first coordinate, and the first coordinate is the three-dimensional coordinate of the pixel closest to the virtual camera between the first pixel and the second pixel in the camera coordinate system. The above-mentioned writing of the data in the pixel corresponding to the third pixel in the corresponding relationship to the third pixel in the second image includes: if the three-dimensional coordinate of the first pixel is the same as the first coordinate, writing the data of the first pixel to the third pixel; if the three-dimensional coordinate of the second pixel is the same as the first coordinate, writing the data of the second pixel to the third pixel.

[0011] That is to say, a correspondence between a pixel point closest to the virtual camera and the same pixel point among multiple pixel points can be established through three-dimensional coordinates in the distance image (such as the third image). Subsequently, before writing data to the above-mentioned same pixel point, the three-dimensional coordinates of the multiple pixel points can be compared with the three-dimensional coordinates corresponding to the above-mentioned same pixel point recorded in the distance image to see if they are the same. If the three-dimensional coordinates of a pixel point among the multiple pixel points are the same as the three-dimensional coordinates corresponding to the above-mentioned same pixel point recorded in the distance image, it indicates that the pixel point is the pixel point closest to the virtual camera, and the data of the pixel point is further written to the above-mentioned same pixel point.

[0012] In a possible design of the first aspect, if the difference between the modulus lengths of the two three-dimensional coordinates is within the first difference, then the two three-dimensional coordinates are the same. The modulus length of the three-dimensional coordinate refers to the square root of the sum of the coordinate values ​​of the three directions of the three-dimensional coordinate.

[0013] In a possible design of the first aspect, before writing data to the third pixel point, the method further includes: initializing the three-dimensional coordinates recorded in the fourth pixel point as the second coordinates, and the coordinate values ​​of each direction in the second coordinates are greater than a preset value. If the three-dimensional coordinates of the first pixel point are smaller than the three-dimensional coordinates recorded in the fourth pixel point, the three-dimensional coordinates of the first pixel point are written into the fourth pixel point. If the three-dimensional coordinates of the second pixel point are smaller than the three-dimensional coordinates recorded in the fourth pixel point, the three-dimensional coordinates of the second pixel point are written into the fourth pixel point.

[0014] It is understandable that the smaller the modulus of the three-dimensional coordinate is, the smaller the three-dimensional coordinate is. At the same time, the smaller the modulus of the three-dimensional coordinate is, the closer the three-dimensional coordinate is to the virtual camera.

[0015] That is to say, a preset coordinate (such as the second coordinate) far away from the virtual camera can be initialized in the pixel point (such as the fourth pixel point) corresponding to the above-mentioned same pixel point in the distance image. Then, the distance of each pixel point from the virtual camera is compared with the coordinate recorded in the distance image (which can be a preset coordinate or a coordinate written later) and the distance of the virtual camera. If the former is smaller, it means that the former is closer to the virtual camera, and the coordinate recorded in the distance image is replaced with the three-dimensional coordinate of the former. In this way, the three-dimensional coordinate of the pixel point closest to the virtual camera among the multiple pixels can be written into the distance image.

[0016] In a possible design of the first aspect, if the three-dimensional coordinates of the first pixel point are greater than or equal to the three-dimensional coordinates recorded in the fourth pixel point, or the three-dimensional coordinates of the second pixel point are greater than or equal to the three-dimensional coordinates recorded in the fourth pixel point, data is not written to the fourth pixel point.

[0017] That is to say, after comparing the distance of each pixel point among multiple pixels from the virtual camera with the coordinates recorded in the distance image (which can be preset coordinates or coordinates written subsequently) and the distance of the virtual camera, if the former is larger, it means that the former is farther away from the virtual camera, then the three-dimensional coordinates recorded in the distance image are kept unchanged to ensure that the three-dimensional coordinates of the pixel points closer to the virtual camera are recorded in the distance image.

[0018] In a possible design of the first aspect, the first image further includes a fifth pixel, and the second image further includes a sixth pixel. It is predicted that the position of the fifth pixel after being moved according to the motion vector of the fifth pixel corresponds to the position of the sixth pixel in the second image, and the positions of other pixels in the first image except the fifth pixel after being moved according to their respective motion vectors do not correspond to the position of the sixth pixel in the second image, and the sixth pixel and the fifth pixel display the same information.

[0019] That is to say, for the one-to-one corresponding pixel points in the first image and the second image (such as the one-to-one corresponding fifth pixel point and the sixth pixel point), the information of the corresponding pixel point in the first image (such as the fifth pixel point) can be presented in the pixel point of the second image (such as the sixth pixel point).

[0020] In a possible design of the first aspect, a distance image can also be used to record the correspondence between the fifth pixel and the sixth pixel. Specifically, the electronic device includes a third image, the pixels in the third image correspond to the pixels in the second image one by one, and the seventh pixel of the third image corresponds to the sixth pixel. Before predicting the second image based on the first image, the method also includes: initializing the three-dimensional coordinates recorded in the seventh pixel as the second coordinate, and the coordinate values ​​in each direction of the second coordinate are greater than the preset value. Determine that the three-dimensional coordinate of the fifth pixel is less than the second coordinate, and write the three-dimensional coordinate of the fifth pixel into the sixth pixel. It can be understood that the second coordinate is far away from the virtual camera, so usually the distance between each pixel in the first image (such as the fifth pixel) and the virtual camera (such as the third distance) will be less than the modulus of the second coordinate. Accordingly, predicting the second image based on the first image includes: determining that the three-dimensional coordinate of the fifth pixel is the same as the three-dimensional coordinate recorded in the seventh pixel, and writing the data of the fifth pixel to the sixth pixel.

[0021] That is to say, according to the three-dimensional coordinates recorded in the distance image, it is also possible to write the data of the pixel point (such as the fifth pixel point) of the real frame image into the only corresponding pixel point (such as the sixth pixel point) in the predicted frame image.

[0022] In a possible design of the first aspect, before rendering the first image, the method further includes: intercepting a rendering instruction of the first image, and obtaining the storage location of the first image, the depth image of the first image, and the projection matrix of the virtual camera from the rendering instruction. After rendering the first image, the method further includes: obtaining the first image, the depth image, and the projection matrix from the storage location. Based on the first image, the depth image, and the projection matrix, the two-dimensional coordinates of the first pixel point and the second pixel point are respectively converted into three-dimensional coordinates.

[0023] That is to say, in addition to obtaining the real frame image, the depth image and projection matrix can be further obtained to convert the two-dimensional coordinates of each pixel point in the real frame image into three-dimensional coordinates, so as to obtain the distance image later.

[0024] In a second aspect, the present application further provides an electronic device, the electronic device comprising a display screen, a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program code, the computer program code comprising computer instructions, when the computer instructions are executed by the processor, the electronic device executes the method in the first aspect and any possible design thereof.

[0025] In a third aspect, the present application provides a chip system, which is applied to an electronic device including a display screen and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method as in the first aspect and any possible design method thereof.

[0026] In a fourth aspect, the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.

[0027] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and any possible design thereof.

[0028] It can be understood that the beneficial effects that can be achieved by the electronic device of the second aspect, the chip system of the third aspect, the computer storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1A schematic diagram of the principle of using frame prediction technology to improve frame rate according to an embodiment of the present application;

[0030] Figure 2 is a schematic diagram of a motion compensation technique;

[0031] Figure 3 A schematic diagram of a scene in which multiple pixels move to the same pixel;

[0032] Figure 4 A schematic diagram of the motion compensation technology provided in the embodiment of the present application;

[0033] Figure 5 A hardware structure diagram of a mobile phone provided in an embodiment of the present application;

[0034] Figure 6 A software architecture diagram of a mobile phone provided in an embodiment of the present application;

[0035] Figure 7 A simplified flowchart of the combination of image rendering and motion compensation provided in an embodiment of the present application;

[0036] Figure 8 A timing interaction diagram for obtaining resources required for motion compensation provided in an embodiment of the present application;

[0037] Fig. 9 A flowchart of obtaining a predicted frame according to acquired resources provided in an embodiment of the present application;

[0038] Fig.10 A schematic diagram of a characteristic vector for calculating the brightness of a pixel provided in an embodiment of the present application;

[0039] Fig.11 A flowchart for generating a location image provided by an embodiment of the present application;

[0040] Fig.12 A flowchart of obtaining a predicted image based on a position image provided in an embodiment of the present application;

[0041] Fig.13 A simplified flowchart of the image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a "or" relationship.

[0043] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0044] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0045] In some specific scenarios, electronic devices need to complete a large amount of image rendering. Typically, in scenarios where game applications, virtual reality (VR) applications, augmented reality (AR) applications, and modeling applications are run in electronic devices, electronic devices are often required to complete a large amount of image rendering. Then, the electronic device displays the rendered image.

[0046] Exemplarily, the electronic device may be installed with the above-mentioned various applications (Application, APP). When the application needs to display an image through the electronic device, the application can send instructions to the electronic device so that the electronic device completes the rendering of the image according to the instructions, and finally displays the rendered image through the display screen.

[0047] In the following, the image processing method provided in the embodiment of the present application is mainly explained by taking the game application as an example.

[0048] With the update and development of game applications, the game frame rate and image resolution are getting higher and higher. For example, the game frame rate changes from 60 frames per second (Frames per second, FPS) to 75FPS and then to 90FPS, and the image resolution changes from 720 lines (Progressive, p) pixels to 1080p pixels. In this way, image rendering must also develop in a more efficient and high-quality direction. For example, electronic devices need to render more frames of high-resolution images per second. At the same time, it also means that the power consumption of electronic devices will face greater challenges. For example, electronic devices need to increase from rendering 60 frames per second to rendering 75 frames per second to meet the change of game frame rate from 60FPS to 75FPS. However, the number of images rendered per second increases by 15 frames, which is an extremely high test for the power consumption of electronic devices.

[0049] On this basis, a technology for predicting another frame of image based on two frames of image (referred to as frame prediction technology) was proposed. Compared with the image rendering method, the frame prediction technology consumes less resources to obtain a frame of image. For example, it takes 10mA of current for an electronic device to obtain a frame of image by rendering, while it takes 5mA of current to obtain a frame of image by frame prediction technology. Therefore, the use of frame prediction technology can reduce the power consumption of electronic devices while meeting the game frame rate.

[0050] See also Figure 1, the electronic device can render 60 frames of images per second, and in the process of rendering these 60 frames of images, the electronic device can also predict 15 frames of images through frame prediction technology. In this way, the electronic device can obtain 75 frames of images in one second, thereby increasing the game frame rate from 60 frames per second to 75 frames per second. For example, the electronic device predicts the (1st) frame of image between the 2nd and 3rd frames of image, predicts the (2nd) frame of image between the 6th and 7th frames of image, predicts the (3rd) frame of image between the 10th and 11th frames of image... and predicts the (15th) frame of image between the 58th and 59th frames of image. Moreover, by combining the method of rendering 60 frames of images and predicting 15 frames of images to obtain 75 frames of images, the power consumption of the electronic device can be reduced compared with the method of obtaining 75 frames of images by rendering.

[0051] A typical frame prediction technology is motion compensation technology. Figure 2 , using motion compensation technology, the electronic device can predict the n0th frame image based on the n-1th frame image and the nth frame image:

[0052] S1. The electronic device determines matching point pairs in the n-1th frame image and the nth frame image.

[0053] It can be understood that a matching point pair refers to pixel points that display the same information in two frames of images. For example, pixel point A in the n-1th frame of image and pixel point B in the nth frame of image both display the highest point of the sail, that is, the same information is the highest point of the sail, and the electronic device can determine that pixel point A in the n-1th frame of image and pixel point B in the nth frame of image are a set of matching point pairs.

[0054] S2. The electronic device calculates the motion vector of each pixel point in the n-1th frame image moving to the n0th frame image.

[0055] For any set of matching point pairs, the electronic device can calculate the motion vector between two pixels in the set of matching point pairs. For example, in the n-1th frame image, the coordinates of pixel point A are (u1, v1), and in the nth frame image, the coordinates of pixel point B are (u2, v2), then the motion vector from pixel point A in the n-1th frame image to pixel point B in the nth frame image is (u2-u1, v2-v1).

[0056] The predicted n0th frame image is the image between the n-1th frame image and the nth frame image, so the motion amplitude between the n-1th frame image and the n0th frame image should be half of the motion amplitude between the n-1th frame image and the nth frame image. Then, the motion vector of each pixel point in the n-1th frame image moving to the n0th frame image should also be half of the motion vector of the matching point pair in the n-1th frame image and the nth frame image.

[0057] Exemplarily, the motion vector of pixel point A in the n-1th frame image moving to pixel point B in the nth frame image is (u2-u1, v2-v1), and the electronic device can convert it to obtain the motion vector of pixel point A in the n-1th frame image moving to the corresponding pixel point in the n0th frame image as ((u2-u1) / 2, (v2-v1) / 2).

[0058] S3. The electronic device predicts the n0th frame image.

[0059] The electronic device may generate the n0th frame image by unidirectional interpolation or bidirectional interpolation. The unidirectional interpolation refers to generating the n0th frame image (also referred to as the second image) based on the n-1th frame image and one of the nth frame images (also referred to as the first image); the bidirectional interpolation refers to generating the n0th frame image based on the n-1th frame image and the nth frame image.

[0060] In the following, the method of generating the n0th frame image based on the n-1th frame image (ie, the first image is the n-1th frame image) in one-way interpolation is mainly described as an example, but the actual implementation is not limited to this.

[0061] For each pixel in the (n-1)th frame image, the electronic device may move the position of the pixel based on the motion vector of the pixel to obtain the position of the corresponding pixel in the (n0)th frame image.

[0062] Exemplarily, the electronic device moves the coordinates (u1, v1) of the pixel point A in the n-1th frame image according to the motion vector ((u2-u1) / 2, (v2-v1) / 2) to obtain the position (u0, v0) of the corresponding target pixel point C in the n0th frame image. That is, u1 is moved by (u2-u1) / 2 to obtain u0, and v1 is moved by (v2-v1) / 2 to obtain v0.

[0063] Then, for each pixel in the n-1th frame image, the electronic device writes the data of the pixel (such as color information) to the position of the corresponding pixel in the n0th frame image, so as to obtain the n0th frame image. Exemplarily, the electronic device writes the data of pixel A in the n-1th frame image to the target pixel C (u0, v0) of the n0th frame image, and the highest point of the sail can be presented in the target pixel C (u0, v0) in the n0th frame image.

[0064] Obviously, the motion compensation technology can be used to predict images based on motion vectors. However, conventional motion compensation technology has the following defects: in the above S3 process, if the electronic device calculates that multiple pixels in the n-1th frame image move to the same pixel in the n0th frame image, the electronic device cannot determine which pixel in the n0th frame image should present information of the same pixel. That is, the electronic device cannot solve the problem of selecting and determining when multiple pixels move to the same pixel.

[0065] See also Figure 3 , pixel A1 and pixel A2 in the n-1th frame image can both move to the target pixel C in the n0th frame image. For example, if the information presented by pixel A1 (such as the highest point of the sail) is closer to the virtual camera (referring to the assumed camera when rendering the n-1th frame image), and the information presented by pixel A2 (such as wild geese) is farther from the virtual camera, then at the target pixel C, the data of pixel A1 should block the data of pixel A2. Therefore, the target pixel C of the n0th frame image should present the information presented by pixel A1.

[0066] If the electronic device processes pixel A1 first, writes the data of pixel A1 to the target pixel C, and then processes pixel A2 and writes the data of pixel A2 to the target pixel C, the data of pixel A2 will overwrite the data of pixel A1, that is, the target pixel C will present the information presented by pixel A2, such as wild geese. This result is wrong. If the electronic device processes pixel A2 first, writes the data of pixel A2 to the target pixel C, and then processes pixel A1 and writes the data of pixel A1 to the target pixel C, the data of pixel A1 will overwrite the data of pixel A2, that is, the target pixel C will present the information presented by pixel A1. This result is correct.

[0067] It can be seen that, by using conventional motion compensation technology, the electronic device cannot ensure that the data in the target pixel point C is written correctly, which may cause the predicted image of the n0th frame to be wrong.

[0068] Based on the above problems, an embodiment of the present application provides an image processing method that can be used in scenes where motion compensation technology is used to predict image frames. Specifically, if there are multiple pixel points in the n-1th frame image (two of which can be called the first pixel point and the second pixel point) that can move to the same pixel point in the n0th frame image (also called the third pixel point) according to their respective motion vectors, the electronic device can determine the pixel point that is closest to the virtual camera among the multiple pixel points. The distance between the pixel point and the virtual camera refers to the distance between the spatial point in the three-dimensional scene where the information presented by the pixel point is located and the virtual camera. Figure 4Taking pixel A1 in the figure as an example, pixel A1 presents the information of A1' on the sphere in the three-dimensional scene, that is, the spatial point A1' of the information presented by pixel A1 in the three-dimensional scene is A1'. Therefore, the distance between pixel A1 and the virtual camera refers to the distance between A1' and the camera (ie, the virtual camera).

[0069] Then, the electronic device writes the data of the pixel point closest to the virtual camera into the same pixel point of the n0th frame image. In this way, the electronic device can ensure the correctness of the information presented by the same pixel point in the n0th frame image.

[0070] by Figure 4 Taking the example that pixel point A1 (i.e., the first pixel point is pixel point A1) and pixel point A2 (i.e., the second pixel point is pixel point A2) in the n-1 frame image as shown, both move to the target pixel point C (i.e., the third pixel point is pixel point C) in the n0 frame image according to their respective motion vectors, in the n-1 frame image, pixel point A1 presents the information of A1' on the sphere in the three-dimensional scene, and pixel point A2 presents the information of A2' on the cube in the three-dimensional scene, and A1' is closer to the camera and A2' is farther from the camera, the electronic device can write the data of pixel point A1 in the n-1 frame image into the target pixel point C in the n0 frame image.

[0071] Furthermore, electronic devices use some conventional frame prediction technologies, which often lead to image quality problems. For example, in some conventional frame prediction technologies, the electronic device generates the n0th frame image by superimposing the n-1th frame image and the nth frame image in a corresponding proportion. However, superposition can cause ghosting. Another example is that in some conventional frame prediction technologies, the electronic device generates the n0th frame image by moving the pixel block in the n-1th frame image or the nth frame image. However, the actual motion vectors of the multiple pixels included in a pixel block are different, and forcing the multiple pixels to move together as a whole with the same displacement is likely to cause distortion. In contrast, the electronic device adopts the solution of the present application, and each pixel in the n-1th frame image moves according to its own motion vector, so the above-mentioned distortion and ghosting problems will not occur.

[0072] It should be noted here that the movement of the pixel points of the n-1th frame image to the pixel points in the n0th frame image mentioned in this article does not mean that the pixel points actually move, but that the position to which the pixel points of the n-1th frame image are predicted to reach after moving according to their motion vectors corresponds exactly to the position of the corresponding pixel points in the n0th frame.

[0073] Exemplarily, the electronic device may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0074] See also Figure 5 , is a hardware structure diagram of an electronic device. Figure 5 As shown, taking the electronic device as a mobile phone as an example, the electronic device may include a processor 210, an external memory interface 220, an internal memory (referred to as memory) 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0075] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0076] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0077] The charging management module 240 is used to receive the charging input of the charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives the input of the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc.

[0078] The wireless communication function of the electronic device can be realized by antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied in electronic devices. Wireless communication module 260 can provide solutions for wireless communication including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied in electronic devices.

[0079] The electronic device can realize the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0080] The electronic device can realize the camera function through the camera 293, ISP, video codec, GPU, display screen 294, application processor AP, neural network processor NPU, etc.

[0081] The electronic device can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0082] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. TM ) system as an example to illustrate the software structure of electronic devices. The layered architecture divides the software system of electronic devices into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces.

[0083] See also Figure 6 Taking mobile phones as an example, the software and hardware architecture of electronic devices can include application (APP) layer, framework layer, system library and hardware layer.

[0084] The application layer may also be referred to as the application program layer. The application program layer may include a series of application packages. The application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. In some embodiments, the application program layer may also include applications that need to display images or videos to users after image rendering. For example, the application program layer may also include game applications, such as and Etc. As another example, the application layer may also include AR applications, VR applications, modeling applications, etc.

[0085] The framework layer can also be called the application framework layer. The framework layer can provide an application programming interface (API) and a programming framework for the application of the application layer. The framework layer includes some predefined functions. Exemplarily, the framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, etc.

[0086] In the embodiment of the present application, one or more functional modules may be provided in the framework layer to implement the image processing method provided in the embodiment of the present application. It is understandable that these functional modules are controlled by the CPU to perform corresponding functions.

[0087] The system library can include multiple functional modules, such as surface manager, media framework, standard C library (libc), SQLite, graphics library, etc.

[0088] The graphics library may include at least one of the following: Open Graphics Library (OpenGL), OpenGL for Embedded Systems (OpenGL ES), Vulkan, etc. The graphics library may provide drawing and operation of 2D graphics and 3D graphics in the application. In some embodiments, the CPU or GPU may call the graphics library to implement image rendering.

[0089] The hardware layer may include a processor, such as a CPU, a GPU, an NPU, etc. Also, the hardware layer may include a memory. In an embodiment of the present application, a frame buffer may be created in the memory for storing rendering results and noise reduction processing results.

[0090] See also Figure 7 In the process of image rendering, the application, CPU, GPU, memory and display screen in the above-mentioned hardware and software structure of the electronic device work as follows: the application can issue a rendering instruction to instruct the rendering of an image. After receiving the rendering instruction, the CPU can call the corresponding graphics drawing application programming interface (Application Programming Interface, API) in the image library to instruct the GPU to perform the rendering operation corresponding to the rendering instruction. The GPU executes the rendering instruction and stores the rendered color image (such as the nth frame image) in the memory. Finally, the color image in the memory can be sent to the display screen for display.

[0091] In some embodiments, the CPU can obtain the storage location of the resources required for motion compensation from the rendering instructions. The resources required for motion compensation include the projection matrix, depth image and color image of the camera. Correspondingly, the storage location includes the texture ID for storing the depth image, the texture ID for storing the color image, and the cache address for storing the projection matrix. The CPU sends the storage location to the GPU. The GPU can obtain the resources required for motion compensation based on the storage location, perform frame prediction, obtain the n0th frame image and store it in the memory. Finally, the n0th frame image in the memory can also be sent to the display screen for display.

[0092] Furthermore, if multiple pixels in the n-1th frame image move to the same pixel in the n0th frame image, the GPU can write the data of the pixel closest to the camera among the multiple pixels into the corresponding pixel of the n0th frame image when performing image prediction, thereby ensuring the correctness of the n0th frame image.

[0093] The image processing method provided in the embodiment of the present application can be implemented in an electronic device having the above-mentioned software and hardware structure.

[0094] In the image processing method provided in the embodiment of the present application, the electronic device can first obtain the resources required for the motion compensation operation. The resources required for the motion compensation operation include the n-1th frame image, the nth frame image, the projection matrix of the virtual camera when rendering the n-1th frame image, and the depth image obtained when rendering the n-1th frame image.

[0095] The electronic device can obtain the storage location of the resources required for the motion compensation operation from the rendering instruction by analyzing the rendering instruction, and then obtain the resources required for the motion compensation operation from the storage location. Exemplarily, the electronic device obtains the storage locations of the rendered n-1 frame image, the projection matrix of the virtual camera when rendering the n-1 frame image, and the depth image obtained when rendering the n-1 frame image from the rendering instruction of the n-1 frame image. Then, the electronic device obtains the n-1 frame image, the projection matrix, and the depth image (which may be referred to as resource 1) from these storage locations. And, the electronic device obtains the storage location of the rendered n frame image from the rendering instruction of the n frame image. Then, the electronic device obtains the n frame image (which may be referred to as resource 2) from the storage location.

[0096] At this point, it should be noted that the n-1th frame image, the n0th frame image and the nth frame image in this article are all color images that do not include user interface (UI) elements. Taking the game scene as an example, the n-1th frame image, the n0th frame image and the nth frame image do not include game interface operation controls (such as skill controls, direction controls, etc.), chat information, etc.

[0097] For details, see Figure 8 The process of the electronic device obtaining the resources required for the above motion compensation operation includes:

[0098] S801. The application sends a rendering instruction 1 for the n-1th frame image to the CPU.

[0099] S802 . The CPU intercepts rendering instruction 1 , and obtains storage location 1 of resource 1 required for motion compensation operation from rendering instruction 1 .

[0100] Among them, resource 1 includes the n-1th frame image, the depth image and the projection matrix.

[0101] Exemplarily, the CPU may intercept rendering instructions through a hook function.

[0102] The rendering instruction carries a storage location, and the CPU can obtain the storage location 1 of the resource 1 from the rendering instruction.

[0103] Exemplarily, the application may specify the texture storing the color image, the depth image and the like by calling the glBindTexture function in the rendering instruction, that is, the storage location of the image is the texture specified by the glBindTexture function.

[0104] As another example, the application may specify a cache for storing virtual camera data (such as the space matrix and projection matrix of the virtual camera) by calling the glBindBuffer function in a rendering instruction, that is, the storage location of the virtual camera data is the cache specified by the glBindBuffer function.

[0105] It should be noted that in the process of rendering a frame of image, the application can send multiple rendering instructions (Draw Command) to the CPU in sequence to achieve different rendering functions. That is to say, the above rendering instruction 1 may include multiple rendering instructions. The CPU will intercept different rendering instructions among the multiple rendering instructions at different times, and analyze the rendering instructions to obtain the storage location of the corresponding resource 1. For example, at time point 1, a rendering instruction for a cache that specifies the storage of virtual camera data is obtained, and the storage location of the virtual camera data is obtained by analysis; at time point 2, a rendering instruction for a texture that specifies the storage of a color image is obtained, and the storage location of the color image is obtained by analysis.

[0106] S803. The CPU sends storage location 1 to the GPU.

[0107] S804. The CPU sends a rendering instruction 1 to the GPU.

[0108] S805 . The GPU performs image rendering based on the rendering instruction 1 , and obtains a rendering result 1 of the n-1th frame image.

[0109] The GPU can obtain rendering results such as the n-1th frame image, the depth image, the shadow image, etc. through rendering. It can be seen that the rendering result 1 includes the n-1th frame image and the depth image in the resource 1.

[0110] At this point, it should be noted that the execution order of the above S802-S803 and S804-S805 is not based on Figure 8 For example, S802-S803 may be after S804 or S805; or, S802 is before S804, and S803 is after S804. It is sufficient as long as S802-S803 is completed before S806 below.

[0111] S806 . The GPU obtains resource 1 based on storage location 1 .

[0112] Exemplarily, the GPU obtains the n-1th frame image from the texture specified by the glBindTexture function, and obtains the projection matrix from the buffer specified by the glBindBuffer function.

[0113] S807: The application sends a rendering instruction 2 of the nth frame image to the CPU.

[0114] S808 . The CPU intercepts rendering instruction 2 , and obtains storage location 2 of resource 2 required for motion compensation operation from rendering instruction 2 .

[0115] For the specific implementation of S808, please refer to the description of S802 above, which will not be repeated here.

[0116] S809. The CPU sends storage location 2 to the GPU.

[0117] S810. The CPU sends a rendering instruction 2 to the GPU.

[0118] S811. The GPU renders an image based on the rendering instruction 2 to obtain a rendering result 2 of the nth frame image.

[0119] The GPU can obtain rendering results such as the n-th frame image, the depth image, the shadow image, etc. through rendering. It can be seen that the rendering result 2 includes the n-th frame image in the resource 2.

[0120] Similarly, the execution order of the above S808-S809 and S810-S811 is not based on Figure 8 For example, S808-S809 may be after S810 or S811; or, S808 is before S810, and S809 is after S811. It is sufficient as long as S810-S811 is completed before S812 below.

[0121] S812. GPU obtains resource 2 based on storage location 2.

[0122] It should be noted that the nth frame image in the process of obtaining the previous predicted frame image (such as recorded as the n1th frame image) may be the n-1th frame image in the process of obtaining the next predicted frame image (such as recorded as the n2th frame image).

[0123] For example, when the n1th frame image between the 1st frame image and the 2nd frame image is predicted, the 1st frame image is the n-1th frame image, and the 2nd frame image is the nth frame image. When the n2th frame image between the 2nd frame image and the 3rd frame image is predicted, the 2nd frame image is the n-1th frame image, and the 3rd frame image is the nth frame image. Obviously, the nth frame image in the previous prediction and the n-1th frame image in the next prediction are the same, both of which are the 2nd frame image.

[0124] At the same time, in each prediction, it is necessary to obtain resource 1 corresponding to the n-1th frame image and resource 2 corresponding to the nth frame image. Then, for the same frame image (such as the second frame image in the previous example), in the previous prediction, the image is used as the nth frame image, and resource 2 corresponding to the image needs to be obtained. In the next prediction, if the image is used as the n-1th frame image, resource 1 corresponding to the image needs to be obtained. In other words, in the two predictions, it may be necessary to obtain resource 2 and resource 1 corresponding to the same frame image respectively.

[0125] Therefore, in order to simplify the resource acquisition process, after receiving the rendering instruction of any frame image, the CPU can obtain the storage location of the color image, depth image and projection matrix from the rendering instruction. Subsequently, the GPU can obtain the color image, depth image and projection matrix of any frame image based on the storage location. In this way, in the previous prediction, the GPU can use the color image, depth image and projection matrix of the nth frame image as resource 2; and in the next prediction, the GPU can use the color image, depth image and projection matrix of the n-1th frame image as resource 1, without having to re-acquire resources for each prediction.

[0126] After obtaining the resources required for motion compensation operation (such as resource 1 and resource 2 mentioned above), the electronic device can use these resources to predict the n0th frame image.

[0127] In the embodiment of the present application, when the electronic device predicts the n0th frame image, if it is determined that multiple pixels in the n-1th frame image will move to the same pixel in the n0th frame image, the electronic device can write the data of the pixel closest to the virtual camera among the multiple pixels into the same pixel in the n0th frame image. In this way, it can be avoided that the information presented by the pixel farther from the virtual camera blocks the information presented by the pixel closer to the virtual camera, and the information presented by the pixel closest to the virtual camera is guaranteed in the n0th frame image, thereby improving the accuracy of the n0th frame image.

[0128] For details, see Fig. 9 , the process of implementing motion compensation operation by an electronic device (such as a GPU in an electronic device) includes:

[0129] S901: GPU determines multiple groups of matching point pairs in the n-1th frame image and the nth frame image.

[0130] The GPU may determine multiple sets of matching points based on the color information of each pixel in the n-1th frame image in resource 1 and the color information of each pixel in the nth frame image in resource 2. The specific implementation of determining the matching point pairs is not specifically limited in the embodiments of the present application.

[0131] In some embodiments, the GPU may determine two pixel points whose brightness difference is within difference 1 and whose brightness vector difference is within difference 2 as a set of matching point pairs. It is understandable that the difference of the vectors may be obtained by subtracting the modulus lengths of the vectors.

[0132] The brightness of a pixel can be obtained by converting the RGB color information of the pixel. For example, the brightness of a pixel is Intensity = Color.r*0.299+Color.g*0.587+Color.b*0.114. Color.x represents the color information of the x channel, and x can be r, g, or b.

[0133] And, the brightness vector refers to a feature sequence formed by the magnitude relationship between the brightness of a pixel and the brightness of multiple adjacent pixels. Therefore, the brightness vector can be obtained by comparing the brightness of a pixel with the brightness of adjacent pixels.

[0134] Calculate Fig.10Taking the brightness vector of the pixel point P shown as an example, pixel point n1, pixel point n2, pixel point n3 and pixel point n4 are all adjacent pixels of pixel point P. The GPU can compare the brightness of pixel point P with pixel point n1, pixel point n2, pixel point n3 and pixel point n4 respectively. If the brightness of pixel point P is higher, the characteristic value 1 is recorded, such as 1; if the brightness of the adjacent pixel point is higher, the characteristic value 2 is recorded, such as 0. In this way, a brightness vector composed of multiple characteristic values ​​corresponding to multiple adjacent pixels can be obtained. For example, the brightness of pixel point P is higher than the brightness of pixel point n1, and 1 is recorded; the brightness of pixel point P is lower than the brightness of pixel point n2, and 0 is recorded; the brightness of pixel point P is lower than the brightness of pixel point n3, and 0 is recorded; the brightness of pixel point P is higher than the brightness of pixel point n4, and 1 is recorded. Thus, the brightness vector Special = (1 0 0 1) can be obtained.

[0135] Exemplarily, the GPU may calculate the brightness vector using the following formula:

[0136] Special=sign(Intensity_P–Intensity_n1)< <n|sign(Intensity_P–Intensity_n2)<<n–1|…

[0137] Among them, Intensity_P is the brightness of the pixel P of the feature vector to be calculated, n is the number of adjacent pixels, and Intensity_ni (i is taken from 1, 2, 3, 4...) is the brightness of the i-th adjacent pixel. sign is a sign operation. If Intensity_P is greater than Intensity_ni, sign takes the eigenvalue 1, and if Intensity_P is less than Intensity_ni, sign takes the eigenvalue 2. << is a left shift operator. Left shifting k (k is taken from n, n-1, n-2...) bits is multiplied by 2 to the kth power. | is an or operation.

[0138] Through the shift operation and OR operation in the above formula, the brightness vector can be obtained efficiently.

[0139] By adopting the above method, the GPU can calculate the brightness and brightness vector of each pixel in the n-1th frame image, and calculate the brightness and brightness vector of each pixel in the nth frame image.

[0140] The GPU can calculate the brightness of any pixel in the n-1th frame image and the brightness of each pixel in the nth frame image, obtain the brightness difference, and compare the brightness difference with difference 1; and the GPU can calculate the brightness vector of the pixel in the n-1th frame image and the brightness vector of each pixel in the nth frame image, obtain the brightness vector difference, and compare the brightness difference with difference 2. Thus, the pixel that forms a matching point pair with the pixel in the n-1th frame image can be determined from the nth frame image.

[0141] By using this embodiment, two pixel points whose brightness difference is not large and whose brightness relationship with adjacent pixel points is also similar can be determined as a matching pixel point pair.

[0142] S902 , the GPU calculates the motion vector of each pixel point in the n-1 th frame image moving to the n0 th frame image.

[0143] Among them, the GPU can calculate the motion vector of each pixel point in the n-1th frame image to the n0th frame image based on the two-dimensional coordinates of each pixel point in the n-1th frame image in resource 1 and the two-dimensional coordinates of each pixel point in the nth frame image, as well as the matching point pairs determined in the above S901.

[0144] For the process of calculating motion vectors, please refer to the previous article Figure 2 The relevant description about S2 is not repeated here.

[0145] In a specific implementation, a motion vector diagram may be used to represent the motion vector of a large number of pixels in the n-1th frame image moving to the n0th frame image. Exemplarily, the pixels in the motion vector diagram correspond one-to-one to the pixels in the n-1th frame image, and the data of each pixel in the motion vector diagram represents the motion vector of the pixel in the n-1th frame image moving to the corresponding pixel in the n0th frame image. For example, the motion vector diagram is a dual-channel image (such as an R channel and a G channel), one channel (such as the R channel) is used to record the displacement of the motion vector in one direction (such as the u direction), and the other channel (such as the G channel) is used to record the displacement of the motion vector in another direction (such as the v direction).

[0146] S903, the GPU determines the correspondence between the pixel points in the n0th frame image and the pixel points in the n-1th frame image. In the correspondence, a pixel point in the n0th frame image corresponds to a pixel point in the n-1th frame image that moves to the pixel point and is closest to the virtual camera.

[0147] The GPU may determine the corresponding relationship according to the two-dimensional coordinates of each pixel point in the n-1th frame image in resource 1, the depth image and the projection matrix, and the motion vector determined in S902 above.

[0148] In the embodiment of the present application, the GPU can not only determine the position of each pixel point in the n-1 frame image moving to the n0 frame image according to the two-dimensional coordinates and motion vector of each pixel point in the n-1 frame image, but also determine the position of each pixel point in the n-1 frame image moving to the n0 frame image according to the two-dimensional coordinates and motion vector of each pixel point in the n-1 frame image. Figure 2 The relevant description of S2 in the above will not be repeated here. In addition, the depth image and the projection matrix can be further combined to solve the problem of selecting and determining the movement of multiple pixels in the n-1th frame image to the same pixel in the nth frame image.

[0149] In some embodiments, the GPU can combine the depth image and the projection matrix to convert the two-dimensional coordinates of the pixel point in the n-1th frame image into the three-dimensional coordinates in the camera coordinate system of the virtual camera. It can be understood that the origin of the camera coordinate system is usually the virtual camera, so the three-dimensional coordinates can be used to measure the distance between the pixel point and the virtual camera. Subsequently, after the GPU moves the pixel point in the n-1th frame image to the corresponding pixel point in the n-1th frame image according to the motion vector, it can determine whether the pixel point has a corresponding relationship with the corresponding pixel point in the n-1th frame image in combination with the three-dimensional coordinates corresponding to the pixel point in the n-1th frame image.

[0150] Among them, one or more pixel points in the (n-1)th frame image can move to a pixel point in the (n0)th frame image according to the motion vector.

[0151] If a pixel point in the n-1th frame image (such as recorded as pixel point A, also called the fifth pixel point) moves to a pixel point in the n0th frame image (such as recorded as target pixel point C, also called the sixth pixel point), that is, the correspondence between the pixels is one-to-one, the GPU can determine that the target pixel point C in the n0th frame image corresponds to the pixel point A in the n-1th frame image.

[0152] If multiple pixels in the n-1th frame image (such as pixel A1 and pixel A2) can move to the target pixel C in the n0th frame image, the GPU can determine that pixel A1 is closer to the virtual camera by combining the 3D coordinates corresponding to pixel A1 and the 3D coordinates corresponding to pixel A2, and thus determine that the target pixel C in the n0th frame image corresponds to pixel A1 in the n-1th frame image. ) is smaller, the closer the pixel is to the virtual camera; the larger the modulus of the three-dimensional coordinate is, the farther the pixel is from the virtual camera.

[0153] Exemplarily, the process of GPU converting the two-dimensional coordinates of the pixel point in the n-1th frame image into the three-dimensional coordinates in the virtual camera coordinate system is as follows:

[0154] The GPU can use the global workgroup coordinates (gl_GlobalInvocationID.xy) of each thread in the GPU's thread workgroup to calculate the normalized coordinates (u, v) of each pixel in the image, and obtain the depth value (Depth0(u, v)) of the pixel corresponding to the normalized coordinate in the depth image.

[0155] The thread work group of the GPU includes multiple threads, and multiple threads can calculate synchronously to improve computing efficiency. In the above process of calculating normalized coordinates and obtaining depth values, each thread can process one pixel to complete the calculation of normalized coordinates and the acquisition of depth values. The global work group coordinates of each thread can be understood as the image coordinates of the pixel point it processes in the n-1 frame image. For example, thread 1 is used to process the pixel point in the 512th row and 512th column of the n-1 frame image, then the global work group coordinates of thread 1 are (512, 512).

[0156] It can be understood that the image coordinate system and the normalized coordinate system are two commonly used plane coordinate systems in the image processing process. Generally, the image coordinate system can be used in the scenario of writing data to the image; and the normalized coordinates can be used in the scenario of image operation to improve the operation efficiency. Each thread can calculate the ratio of the coordinate value of the x-axis in the image coordinate of the pixel point in the image coordinate system of the n-1th frame image to the number of pixels included in the x-axis direction of the n-1th frame image, and obtain the value of u in the normalized coordinate of the pixel point; and calculate the ratio of the coordinate value of the y-axis in the image coordinate of the pixel point in the image coordinate system of the n-1th frame image to the number of pixels included in the y-axis direction of the n-1th frame image, and obtain the value of v in the normalized coordinate of the pixel point. Regarding the image coordinate system, normalized coordinates, and the conversion between the two, this article will not go into too much explanation.

[0157] The depth image corresponds to the pixels in the n-1th frame image one by one, and each thread can obtain the depth value of the pixel it processes from the depth image. In a specific implementation, the thread can efficiently obtain the depth value from the depth image through the sampling pipeline of the GPU.

[0158] After obtaining the depth value of each pixel in the n-1th frame image, the GPU can convert the normalized coordinates of the pixel, the depth value and the projection matrix to obtain the three-dimensional coordinates. For example, the following formula can be used to convert the three-dimensional coordinates:

[0159] tempData.xyzw=[(u,v)*2.0–1.0, Depth0(u,v),1.0]*Inverse(projectionMatrix);

[0160] viewPos.xyz=(tempData.x / tempData.w,tempData.y / tempData.w,tempData.z / tempData.w);

[0161] Among them, tempData.xyzw is the homogeneous coordinate of the pixel in the Normalized Device Coordinate (NDC) system, projectionMatrix is ​​the projection matrix, and Inverse is the inverse operation. projectionMatrix can be used to convert three-dimensional coordinates to two-dimensional coordinates. Therefore, when converting two-dimensional coordinates to three-dimensional coordinates, the inverse matrix of projectionMatrix can be used for conversion. viewPos.xyz is the converted three-dimensional coordinate, and tempData.j (j is taken from x, y, z, w) represents the coordinate value of the j axis in the homogeneous coordinate.

[0162] S904 . The GPU writes the data of the pixel points in the n-1th frame image into the corresponding target pixel points in the n0th frame image according to the corresponding relationship.

[0163] Exemplarily, in the corresponding relationship, pixel point A1 of the n-1th frame image corresponds to target pixel point C of the n0th frame image, and the GPU can write the data of pixel point A1 in the n-1th frame image to the target pixel point C of the n0th frame image.

[0164] Since, in the corresponding relationship, a pixel point in the n0th frame image corresponds to a pixel point in the n-1th frame image that moves to the pixel point and is closest to the virtual camera, it is possible to avoid presenting obscured information in the n0th frame image, thereby improving the accuracy of the n0th frame image.

[0165] In some embodiments, the GPU may represent the correspondence between the pixel points in the n-1th frame image and the pixel points in the n0th frame image through a position image. Fig.11 In this embodiment, S903 further includes the following steps:

[0166] S1101, GPU initializes position image.

[0167] A pixel in the position image corresponds to a pixel in the n0th frame image. A pixel in the position image (such as the target pixel C, which can also be called the fourth pixel) is used to record the three-dimensional coordinates, which refer to the three-dimensional coordinates of the pixel that moves from the n-1th frame image to the target pixel C (i.e., the third pixel) in the n0th frame image and is closest to the virtual camera. Since the three-dimensional coordinates have three values, the position image can be a three-channel image (such as RGB channels), and one channel is used to record the coordinate value of one direction.

[0168] When initializing the position image, a larger initial value (such as greater than a preset value) is set for each pixel in the position image. The initial three-dimensional coordinates (also called second coordinates) of each pixel are far away from the virtual camera. For example, the initial value of each channel is set to 0xFF.

[0169] Subsequently, for each pixel point in the n-1th frame image (which can be recorded as the current pixel point), the GPU can determine that the current pixel point moves to the target pixel point in the n0th frame image, and compare the three-dimensional coordinates corresponding to the current pixel point with the three-dimensional coordinates recorded in the target pixel point of the position image. If the three-dimensional coordinates of the current pixel point are smaller, the three-dimensional coordinates of the pixel point are written into the target pixel point of the position image. After all the pixels in the n-1th frame image are processed, the final updated position image can be obtained. Specifically, as shown in S1102-S1107 below:

[0170] S1102 , the GPU determines that the current pixel point A2 in the n-1th frame image moves to the target pixel point C in the n0th frame image.

[0171] The GPU can obtain the target pixel C in the n0th frame image from the current pixel A2 based on the two-dimensional coordinates (such as normalized coordinates) of the current pixel A2 in the n-1th frame image and the motion vector of the current pixel A2. For details, please refer to the previous article Figure 2 The relevant description of S3 is not repeated here.

[0172] S1103 . The GPU compares the modulus of the three-dimensional coordinate 1 of the current pixel point A2 with the modulus of the initial three-dimensional coordinate recorded in the target pixel point C in the position image.

[0173] Exemplarily, the three-dimensional coordinate 1 is (x2, y2, z2), and the GPU locates the target pixel point C in the position image according to the two-dimensional coordinate of the target pixel point C, thereby obtaining the initial three-dimensional coordinate of the target pixel point C as (0xFF, 0xFF, 0xFF). The GPU can compare the modulus length of (x2, y2, z2) with the modulus length of (0xFF, 0xFF, 0xFF).

[0174] S1104 , the GPU determines that the modulus length of the three-dimensional coordinate 1 is smaller than the modulus length of the initial three-dimensional coordinate, writes the three-dimensional coordinate 1 into the target pixel point C in the position image, and obtains an updated position image.

[0175] Usually, the initial coordinate values ​​are already large, so the modulus of the three-dimensional coordinate 1 can be smaller. For example, the modulus of (x2, y2, z2) is smaller than the modulus of (0xFF, 0xFF, 0xFF).

[0176] The GPU writes the 3D coordinate 1 to the target pixel C of the position image, and can write the 3D coordinate of closer information to the position image. For example, (x2, y2, z2) is written to the pixel C in the position image, overwriting the initial 3D coordinate (0xFF, 0xFF, 0xFF).

[0177] S1105 , the GPU determines that the current pixel point A1 in the n-1th frame image moves to the target pixel point C in the n0th frame image.

[0178] Similar to S1102 above, the GPU can obtain the target pixel C in the n0th frame image after the current pixel A1 moves based on the two-dimensional coordinates of the current pixel A1 in the n-1th frame image and the motion vector of the current pixel A1. For details, please refer to the above Figure 2 The relevant description of S3 is not repeated here.

[0179] S1106 . The GPU compares the modulus length of the three-dimensional coordinate 2 of the current pixel point A1 with the modulus length of the updated three-dimensional coordinate recorded in the target pixel point C in the position image.

[0180] For example, the three-dimensional coordinate 2 is (x1, y1, z1), and the updated three-dimensional coordinate is (x2, y2, z2). The GPU can compare the modulus of (x1, y1, z1) with the modulus of (x2, y2, z2).

[0181] S1107 . The GPU determines that the modulus length of the three-dimensional coordinate 2 is smaller than the modulus length of the updated three-dimensional coordinate, writes the three-dimensional coordinate 2 into the target pixel point C in the position image, and obtains the updated position image.

[0182] If the 3D coordinate 2 is smaller than the updated 3D coordinate, it means that the distance between the current pixel A1 in the n-1th frame image and the virtual camera is closer. Then, by writing the 3D coordinate 2 into the target pixel C in the position image, the 3D coordinate of the closer information can be written into the position image. For example, writing (x1, y1, z1) into the pixel C in the position image overwrites the 3D coordinate (x2, y2, z2).

[0183] It can be understood that if the three-dimensional coordinates of the current pixel point (such as three-dimensional coordinates 2) are greater than or equal to the updated three-dimensional coordinates, it means that the distance between the current pixel point in the n-1th frame image and the virtual camera is farther or equal, so the three-dimensional coordinates currently recorded in the position image can be kept unchanged.

[0184] It can be understood that if there are still pixels in the n-1th frame image that can move to the target pixel point C of the n0th frame image, the above S1105-S1107 can be repeatedly executed to write the three-dimensional coordinates of the pixel point closest to the virtual camera among all the multiple pixels that can move to the target pixel point C into the target pixel point C.

[0185] If only the current pixel point A2 (also called the fifth pixel point) in the n-1th frame image can move to the target pixel point C (also called the sixth pixel point) in the n0th frame image, then for the target pixel point C in the n0th frame image, there is no subsequent S1105-S1107. Accordingly, the updated three-dimensional coordinates of the target pixel point C will remain unchanged as the three-dimensional coordinates 1.

[0186] In practice, the atomic operation of the GPU can be used to compare and obtain the smaller value of the three-dimensional coordinates and write it into the position image. That is, the above S1103-S1104 and S1106-S1107 can be completed by using the Atomic operation of the GPU.

[0187] The three-dimensional coordinates are recorded in each pixel of the finally updated position image. At the same time, the three-dimensional coordinates correspond to the two-dimensional coordinates of the pixel in the n-1th frame image, and the pixel in the position image corresponds to the pixel in the n0th frame image. Therefore, recording the three-dimensional coordinates of each pixel in the position image is equivalent to obtaining the correspondence between the pixel in the n-1th frame image and the pixel in the n0th frame image.

[0188] Exemplarily, the three-dimensional coordinates of pixel point A1 of the n-1th frame image are recorded in the target pixel point C of the position image, and the corresponding relationship includes: pixel point A1 of the n-1th frame image corresponds to pixel point C of the n0th frame image.

[0189] And, in this embodiment, in S904, for each pixel point in the n-1th frame image (also recorded as the current pixel point), the GPU can compare the three-dimensional coordinates of the current pixel point with the three-dimensional coordinates recorded in the target pixel point of the position image; if the two are the same, the GPU writes the data of the current pixel point to the target pixel point in the n0th frame image; if the two are different, the GPU does not write data to the n0th frame image. In this way, the GPU can ensure that data is written accurately according to the corresponding relationship.

[0190] See also Fig.12 , S904 further includes the following steps:

[0191] S1201. GPU determines that the current pixel point A2 in the n-1th frame image moves to the target pixel point C in the n0th frame image.

[0192] S1202, the GPU determines that the three-dimensional coordinate 1 corresponding to the current pixel point A2 is different from the three-dimensional coordinate 3 recorded in the target pixel point C in the finally updated position image.

[0193] The three-dimensional coordinate 3 may also be referred to as the first coordinate.

[0194] In a specific implementation, the GPU may calculate modulus 1 of 3D coordinate 1 and modulus 2 of 3D coordinate 3, respectively, and then calculate the difference between modulus 1 and modulus 2. If the difference is close to 0, it is determined that 3D coordinate 1 and 3D coordinate 3 are the same.

[0195] In another specific implementation, the GPU may respectively calculate the difference 1 of the coordinate values ​​of the three-dimensional coordinate 1 and the three-dimensional coordinate 3 in the x-axis direction, the difference 2 of the coordinate values ​​in the y-axis direction, and the difference 3 of the coordinate values ​​in the z-axis direction. If the difference 1, the difference 2, and the difference 3 are all close to 0, it is determined that the three-dimensional coordinate 1 and the three-dimensional coordinate 3 are the same.

[0196] If the three-dimensional coordinates 1 and 3 are different, it means that in the correspondence, the target pixel point C of the n0th frame image does not correspond to the current pixel point A2 in the n-1th frame image, and the GPU does not need to write the data of the current pixel point A2 into the target pixel point C of the n0th frame image.

[0197] For example, the three-dimensional coordinate 1 is (x2, y2, z2), and the three-dimensional coordinate 3 is (x1, y1, z1). The two are different, and the GPU does not need to write data.

[0198] On the contrary, if the three-dimensional coordinates 1 and 3 are the same, it means that in the correspondence, the target pixel point C of the n0th frame image corresponds to the current pixel point A2 in the n-1th frame image, and the GPU can write the data of the current pixel point A2 into the target pixel point C of the n0th frame image.

[0199] However, in this article, the example of the difference between the three-dimensional coordinate 1 and the three-dimensional coordinate 3 is mainly used for explanation. The GPU continues to process other pixel points in the n-1 frame image. Exemplarily, the current pixel point A1 is processed as follows S1203-S1205:

[0200] S1203 , the GPU determines that the current pixel point A1 in the n-1th frame image moves to the target pixel point C in the n0th frame image.

[0201] S1204 , the GPU determines that the three-dimensional coordinate 2 and the three-dimensional coordinate 3 of the current pixel point A1 are the same.

[0202] A step similar to S1202 may be adopted to obtain a determination result that the three-dimensional coordinate 2 corresponding to the current pixel point A1 is the same as the three-dimensional coordinate 3.

[0203] S1205 . The GPU writes the data of the current pixel point A1 into the target pixel point C of the n0th frame image.

[0204] If the three-dimensional coordinates 2 and 3 are different, it means that in the corresponding relationship, the target pixel point C of the n0th frame image corresponds to the current pixel point A2 in the n-1th frame image. The GPU writes the data of the current pixel point A1 into the target pixel point C of the n0th frame image, and can write the data of the pixel point corresponding to the target pixel point C in the n0th frame image indicated by the corresponding relationship into the target pixel point C.

[0205] Exemplarily, the three-dimensional coordinate 2 is (x1, y1, z1), and the three-dimensional coordinate 3 is (x1, y1, z1), which are the same. The GPU can write the data of the current pixel point A1 in the n-1th frame image to the target pixel point C of the n0th frame image, such as drawing the highest point of the sail to the target pixel point C of the n0th frame image.

[0206] In this embodiment, by adopting the position image method, the three-dimensional coordinates of the information to be presented by each pixel point in the n0th frame image can be first recorded in the position image, and then the pixel points in the n-1th frame image which is the source of the information to be presented by each pixel in the n0th frame image can be accurately determined based on the three-dimensional coordinates recorded in the position image to obtain the accurate n0th frame image.

[0207] In the above Fig.11 In the embodiment of the present invention, the GPU needs to write the three-dimensional coordinates of each pixel point in the n-1 frame image into each pixel point in the distance image, and, in the above Fig.12 In the embodiment of the present invention, the electronic device needs to write the data of each pixel in the n-1th frame image into the corresponding pixel in the n0th frame image. In other words, the GPU needs to realize flexible writing of data to any position of the image. At the same time, compared with the fragment shader: the compute shader can write data to any position of the image more flexibly. Therefore, the GPU can use the compute shader to complete the calculation of the three-dimensional coordinates of each pixel in the n-1th frame image, write the calculated three-dimensional coordinates into the position image, and write the data of each pixel in the n-1th frame image into the corresponding pixel of the n0th frame image.

[0208] Combine the following Fig.13A simplified flow chart briefly describes the work of the CPU and GPU in a complete implementation method provided by this application:

[0209] S1301. The CPU intercepts a rendering instruction.

[0210] The CPU can obtain the storage location of resources by intercepting rendering instructions.

[0211] S1302: GPU obtains resources.

[0212] The GPU can obtain corresponding resources from the storage location obtained from the CPU, such as the n-1th frame image, the depth image, the projection matrix, and the nth frame image.

[0213] Regarding the above S1301-S1302, please refer to the description of S801-S812 above for details.

[0214] S1303: GPU calculates motion vectors. For details, please refer to the description of S902 above.

[0215] The GPU can obtain a motion vector map by calculating the motion vector.

[0216] S1304. The GPU registers the three-dimensional coordinates of the pixel point closest to the camera.

[0217] That is, the GPU records the three-dimensional coordinates of the pixel point closest to the camera corresponding to each pixel point in the distance image, and obtains the final updated distance image. For details, please refer to the process of calculating the three-dimensional coordinates in S903 above and Fig.11 Related instructions.

[0218] S1305, GPU moves pixels.

[0219] The GPU moves each pixel in the n-1th frame image based on the motion vector diagram to obtain the target pixel in the n0th frame image.

[0220] S1306. The GPU determines whether the three-dimensional coordinates are the same.

[0221] That is, the GPU determines whether the three-dimensional coordinates recorded in the pixel point corresponding to the target pixel point in the distance image are the same as the three-dimensional coordinates of the currently moving pixel point in the (n-1)th frame image.

[0222] S1307: If the three-dimensional coordinates are the same, the GPU writes data to the n0th frame image.

[0223] Specifically, the GPU writes the data of the currently moving pixel in the n-1th frame image to the target pixel in the n0th frame image.

[0224] On the contrary, if the three-dimensional coordinates are different, S1305 - S1307 are continued to be executed, and other pixel points in the (n-1)th frame image are processed until all pixel points are processed.

[0225] For the specific implementation of S1305-S1307 above, please refer to the previous article Fig.12 Description of the illustrated embodiment.

[0226] At this point, it should be noted that in the above embodiments, after the motion vector is obtained, the n0th frame image is predicted based on the n-1th frame image. In practice, the n0th frame image can also be predicted based on the nth frame image, or the n0th frame image can be predicted using a two-term interpolation method (i.e., based on the n-1th frame image and the nth frame image). The principles of these two prediction methods are also similar, and the following only briefly describes the differences between these two methods and the above embodiments.

[0227] Method 1: predicting the n0th frame image based on the nth frame image.

[0228] In the first method, the resources required for the motion compensation operation are mainly the resources obtained when rendering the n-th frame image, such as the n-th frame image, the depth image obtained when rendering the n-th frame image, and the projection matrix when rendering the n-th frame image.

[0229] Accordingly, the above S903 can be replaced by S903': the GPU obtains the correspondence between the pixel points in the n0th frame image and the pixel points in the nth frame image according to the nth frame image, the depth image obtained when rendering the nth frame image, the projection matrix when rendering the nth frame image, and the motion vector. Among them, a target pixel point in the n0th frame image corresponds to a pixel point in the nth frame image that moves to the target pixel point and is closest to the virtual camera.

[0230] S904 may be replaced by S904': the GPU writes the data of the pixel points in the nth frame image into the corresponding target pixel points in the nth frame image according to the correspondence between the pixel points in the nth frame image and the pixel points in the nth frame image.

[0231] Method 2: predicting the n0th frame image based on the n-1th frame image and the nth frame image.

[0232] In method 2, the resources required for motion compensation operations include both the resources obtained when rendering the n-1th frame image, such as the n-1th frame image, the depth image obtained when rendering the n-1th frame image, and the projection matrix when rendering the n-1th frame image; and the resources obtained when rendering the nth frame image, such as the nth frame image, the depth image obtained when rendering the nth frame image, and the projection matrix when rendering the nth frame image.

[0233] Accordingly, based on the above S903, the above S903' is also included. Subsequently, based on S904, the above S904' is also included. Finally, the GPU merges the data written in S904 and S904', for example, calculates the average, and finally obtains the n0th frame image.

[0234] The present application also provides an electronic device, which may include: a memory and one or more processors (such as a CPU, a GPU, an NPU, etc.). The memory is coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device may perform the various functions or steps performed by the device in the above method embodiment.

[0235] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0236] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0237] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0238] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the image processing method in the above-mentioned method embodiments.

[0239] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0240] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0242] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0243] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Rendering a first image, wherein the first image includes a first pixel and a second pixel; Predicting a second image based on the first image, the second image including a third pixel; wherein the positions of the first pixel and the second pixel after being moved according to their respective motion vectors are predicted to correspond to the position of the third pixel in the second image, the motion vector being obtained based on at least two frames of rendered images, the at least two frames of images including the first image; The first image and the second image are displayed respectively; wherein, if the first distance between the first pixel and the virtual camera when rendering the first image is less than the second distance between the second pixel and the virtual camera, the third pixel and the first pixel display the same information; if the first distance is greater than the second distance, the third pixel and the second pixel display the same information; the distance between the pixel and the virtual camera includes: the distance between the three-dimensional coordinates of the pixel in the camera coordinate system of the virtual camera and the virtual camera.

2. The method according to claim 1, characterized in that: The method is applied to an electronic device, wherein the electronic device includes a correspondence relationship between pixels in the first image and pixels in the second image, wherein the third pixel in the second image corresponds to a pixel that is closest to the virtual camera between the first pixel and the second pixel; The predicting the second image based on the first image comprises: The data in the pixel point corresponding to the third pixel point in the corresponding relationship is written into the third pixel point.

3. The method according to claim 2, characterized in that The electronic device uses a third image to represent the corresponding relationship, pixels in the third image correspond to pixels in the second image one-to-one, a fourth pixel in the third image corresponds to the third pixel, a first coordinate is recorded in the fourth pixel, and the first coordinate is a three-dimensional coordinate of a pixel that is closest to the virtual camera between the first pixel and the second pixel in the camera coordinate system; The step of writing the data in the pixel point corresponding to the third pixel point in the corresponding relationship into the third pixel point in the second image includes: If the three-dimensional coordinates of the first pixel point are the same as the first coordinates, writing the data of the first pixel point into the third pixel point; If the three-dimensional coordinates of the second pixel point are the same as the first coordinates, the data of the second pixel point is written into the third pixel point.

4. The method according to claim 3, characterized in that: If the difference between the moduli of two three-dimensional coordinates is within the first difference, the two three-dimensional coordinates are the same.

5. The method according to claim 3 or 4, characterized in that: Before writing data to the third pixel, the method further includes: Initializing the three-dimensional coordinates recorded in the fourth pixel point as second coordinates, wherein coordinate values ​​in each direction of the second coordinates are greater than a preset value; If the three-dimensional coordinate of the first pixel point is smaller than the three-dimensional coordinate recorded in the fourth pixel point, writing the three-dimensional coordinate of the first pixel point into the fourth pixel point; If the three-dimensional coordinate of the second pixel point is smaller than the three-dimensional coordinate recorded in the fourth pixel point, the three-dimensional coordinate of the second pixel point is written into the fourth pixel point.

6. The method according to claim 5, characterized in that If the three-dimensional coordinates of the first pixel point are greater than or equal to the three-dimensional coordinates recorded in the fourth pixel point, or the three-dimensional coordinates of the second pixel point are greater than or equal to the three-dimensional coordinates recorded in the fourth pixel point, data is not written to the fourth pixel point.

7. The method according to any one of claims 1 to 6, characterized in that The first image further includes a fifth pixel point, and the second image further includes a sixth pixel point; Among them, it is predicted that the position of the fifth pixel after moving according to the motion vector of the fifth pixel corresponds to the position of the sixth pixel in the second image, and the positions of other pixels in the first image except the fifth pixel after moving according to their respective motion vectors do not correspond to the position of the sixth pixel in the second image, and the sixth pixel and the fifth pixel display the same information.

8. The method according to claim 7, characterized in that The method is applied to an electronic device, wherein the electronic device includes a third image, pixels in the third image correspond to pixels in the second image one-to-one, and a seventh pixel in the third image corresponds to the sixth pixel; Before predicting the second image based on the first image, the method further includes: Initializing the three-dimensional coordinates recorded in the seventh pixel point as second coordinates, wherein coordinate values ​​in each direction of the second coordinates are greater than a preset value; Determine that the three-dimensional coordinate of the fifth pixel point is less than the second coordinate, and write the three-dimensional coordinate of the fifth pixel point into the sixth pixel point; The predicting the second image based on the first image comprises: It is determined that the three-dimensional coordinates of the fifth pixel point are the same as the three-dimensional coordinates recorded in the seventh pixel point, and the data of the fifth pixel point is written into the sixth pixel point.

9. The method according to any one of claims 3 to 6, characterized in that: Before rendering the first image, the method further includes: intercepting a rendering instruction of the first image, and obtaining a storage location of the first image, a depth image of the first image, and a projection matrix of the virtual camera from the rendering instruction; After rendering the first image, the method further includes: Retrieve the first image, the depth image, and the projection matrix from the storage location; The two-dimensional coordinates of the first pixel point and the second pixel point are respectively converted into three-dimensional coordinates based on the first image, the depth image and the projection matrix.

10. An electronic device, characterized in that: The electronic device includes a display screen, a memory and a processor, wherein the display screen and the memory are coupled to the processor; wherein computer instructions are stored in the memory, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 9.

Images