Image generation method, electronic equipment and computer readable storage medium

Abstract

The invention provides an image generation method, electronic equipment and a computer readable storage medium. The image generation method comprises the following steps: determining static pixel points and dynamic pixel points in a plurality of second pixel points according to first coordinates of a plurality of first pixel points in first rendering information and second coordinates of a plurality of second pixel points in second rendering information; determining a third color value of a third static pixel point according to the second color value of the second static pixel point, wherein the third static pixel point corresponds to the second static pixel point; according to the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point, a fourth color value of a third dynamic pixel point is determined, the coordinates of the first dynamic pixel point and the second dynamic pixel point are the same, and the coordinates of the third dynamic pixel point and the second dynamic pixel point are the same; and generating a target image according to the third color value of the third static pixel point and the fourth color value of the third dynamic pixel point, thereby improving the image quality of the target image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image generation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] High-frame-rate games can provide users with a smoother experience, but this increased frame rate also comes with the added problem of high power consumption. To reduce power consumption, frame prediction technology can be used to generate interpolated images. Existing frame prediction technology generally uses image deformation algorithms to generate interpolated images. Image deformation algorithms process the entire image according to specific rules to generate interpolated images, which can easily cause object distortion and poor visual smoothness, thus affecting the quality of the interpolated images. Summary of the Invention

[0003] The present application provides an image generation method, an electronic device, and a computer-readable storage medium, which can improve the image quality of interpolated images.

[0004] To achieve the above objectives, this application adopts the following technical solutions:

[0005] In a first aspect, a method for generating an image is provided, comprising:

[0006] Obtaining first rendering information at a first moment, the first rendering information including a plurality of first color values and a plurality of first coordinates of a plurality of first pixels, wherein the plurality of first pixels, the plurality of first color values, and the plurality of first coordinates correspond to each other in a one-to-one manner;

[0007] Obtaining second rendering information at a second moment, the second rendering information including a plurality of second color values and a plurality of second coordinates of a plurality of second pixels, the plurality of second pixels, the plurality of second color values, and the plurality of second coordinates corresponding one-to-one, and the plurality of first pixels and the plurality of second pixels corresponding one-to-one;

[0008] Determining a plurality of static pixel points and a plurality of dynamic pixel points among the plurality of second pixel points according to the plurality of first coordinates of the plurality of first pixel points and the plurality of second coordinates of the plurality of second pixel points;

[0009] determining a third color value of a third static pixel point according to the second color value of the second static pixel point, where the second static pixel point is any one of the plurality of static pixel points, and the third static pixel point is a pixel point among the plurality of third pixels points corresponding to the second static pixel point;

[0010] Determining a fourth color value of a third dynamic pixel point based on a first color value of a first dynamic pixel point and a second color value of a second dynamic pixel point, where the second dynamic pixel point is any one of the plurality of dynamic pixel points, the first dynamic pixel point is a pixel point in the plurality of first pixel points having the same coordinates as the second dynamic pixel point, and the third dynamic pixel point is a pixel point in the plurality of third pixel points having the same coordinates as the second dynamic pixel point;

[0011] A target image is generated according to the third color value of the third static pixel and the fourth color value of the third dynamic pixel.

[0012] In the above embodiment, by dividing the second pixel points corresponding to the second rendering information into dynamic pixel points of dynamic objects and static pixel points of static objects, for any second static pixel point among the multiple static pixel points, the third color value of the third pixel point corresponding to the second static pixel point in the target image is determined according to the color value of the second static pixel point. Compared with the image deformation method in which the color values of multiple adjacent pixels are determined by the same method, the present application can avoid the object distortion problem that occurs when the depths of adjacent pixels are discontinuous. For any second dynamic pixel point among the multiple dynamic pixel points, the third color value of the third pixel point with the same coordinates as the second dynamic pixel point in the target image is determined according to the color value of the second dynamic pixel point and the first color value of the first dynamic pixel point with the same coordinates as the second dynamic pixel point, so that the third color value of the third pixel point can be close to the second color value of the second dynamic pixel point or the first color value of the first dynamic pixel point. Pixels with the same coordinates correspond to the same position on the display interface. The third color value of the third pixel is close to the color of the first dynamic pixel or the second dynamic pixel at the same position on the display interface. The persistence effect of the human eye can be utilized. When the object moves at a fast speed, it is not easy for the corresponding position to suddenly change color. This can improve the image quality of the target image and thus improve the overall smoothness of the animation.

[0013] In one embodiment, the second rendering information further includes a plurality of second depth values corresponding to the plurality of static pixels, and determining the third color value of the third static pixel according to the second color value of the second static pixel includes:

[0014] Acquire a depth reference map, the depth reference map including a plurality of reference pixels, the plurality of reference pixels corresponding to a plurality of reference depth values in a one-to-one manner, and the plurality of reference pixels corresponding to the plurality of third pixels in a one-to-one manner;

[0015] Obtaining a second depth value, a second color value, and a first motion vector of the second static pixel, where the first motion vector is related to the second motion vector, the second motion vector representing an offset position of a first coordinate of the first static pixel relative to a second coordinate of the second static pixel, the first static pixel being a pixel corresponding to the second static pixel among the plurality of first pixels;

[0016] determining a third coordinate of the third static pixel point according to the second coordinate of the second static pixel point and the first motion vector;

[0017] If the reference depth value corresponding to the third static pixel is equal to the second depth value of the second static pixel, the second color value of the second static pixel is used as the third color value of the third static pixel, wherein the reference depth value corresponding to the third static pixel is the reference depth value of the reference pixel having the same coordinates as the third coordinate of the third static pixel.

[0018] In the above embodiment, based on the motion vector corresponding to each static pixel point, the third color value of the third pixel point corresponding to the static pixel point is determined one by one. Compared with determining the color values of multiple pixel points at the same time, this can avoid the problem of image distortion of static objects in the animation scene in the target image, thereby improving the image quality of the obtained target image.

[0019] In one embodiment, obtaining a depth reference map includes:

[0020] Acquire an initial depth reference map, the initial depth reference map including a plurality of initial pixel points, the plurality of initial pixel points corresponding one-to-one to a plurality of initial depth values, and the plurality of initial pixel points corresponding one-to-one to the plurality of second pixel points;

[0021] performing an update operation on the initial pixel points corresponding to the plurality of static pixel points among the plurality of initial pixel points to obtain a plurality of update values;

[0022] updating the initial depth reference map according to the multiple update values to obtain the depth reference map;

[0023] The update operation includes:

[0024] Obtaining a second depth value and a third motion vector of a fourth static pixel, where the fourth static pixel is any one of the plurality of static pixels, the third motion vector is related to the fourth motion vector, and the fourth motion vector represents an offset position of a first coordinate of a fifth static pixel relative to the second coordinate of the fourth static pixel, where the fifth static pixel is a pixel corresponding to the fourth static pixel among the plurality of first pixels;

[0025] Determining a first initial pixel point according to the second coordinate of the fourth static pixel point and the third motion vector, where the first initial pixel point is a pixel point corresponding to the fourth static pixel point among the multiple initial pixel points;

[0026] If the second depth value of the fourth static pixel is less than the initial depth value corresponding to the first initial pixel, use the second depth value of the fourth static pixel as the update value corresponding to the first initial pixel;

[0027] If the second depth value of the fourth static pixel is greater than or equal to the initial depth value corresponding to the first initial pixel, the initial depth value corresponding to the first initial pixel is used as the update value corresponding to the first initial pixel.

[0028] In the above embodiment, the reference depth value corresponding to each reference pixel point in the depth reference map is the minimum depth value after the corresponding static pixel point is offset, which can reflect the depth value displayed on the display interface when each static pixel point is offset to the corresponding third pixel point. By guiding the third color value of the third pixel point through the depth reference map, the accuracy of the obtained third color value can be improved.

[0029] In one embodiment, before obtaining the second depth value, the second color value, and the first motion vector of the second static pixel, the method further includes:

[0030] generating the second motion vector according to a difference between a first coordinate of the first static pixel and a second coordinate of the second static pixel;

[0031] A half of the second motion vector is used as the first motion vector.

[0032] In the above embodiment, the first motion vector of the pixel point in the target image relative to the static pixel point is determined based on the second motion vector of the first static pixel point and the second static pixel point, so that the static pixel point corresponding to the third pixel point in the target image can be predicted based on the first motion vector, and the third color value of the third pixel point can be determined.

[0033] In one embodiment, determining the fourth color value of the third dynamic pixel point according to the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point includes:

[0034] Obtaining a first color value and a first identifier of the first dynamic pixel point, where the first identifier indicates whether the first dynamic pixel point is a pixel point of a person;

[0035] Obtaining a second color value and a second identifier of the second dynamic pixel point, where the second identifier indicates whether the second dynamic pixel point is a pixel point of a person;

[0036] If the first identifier represents that the first dynamic pixel point is a pixel point of a person, and the second identifier represents that the second dynamic pixel point is a pixel point of a person, the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weightedly summed according to the first weight coefficient and the second weight coefficient to obtain the fourth color value of the third dynamic pixel point; the first weight coefficient corresponds to the first color value of the first dynamic pixel point, and the second weight coefficient corresponds to the second color value of the second dynamic pixel point.

[0037] In one embodiment, the method further comprises:

[0038] If the first identifier indicates that the first dynamic pixel point is not a pixel point of a person, or if the second identifier indicates that the second dynamic pixel point is not a pixel point of a person, obtaining a first depth value of the first dynamic pixel point and obtaining a second depth value of the second dynamic pixel point;

[0039] The first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weightedly summed according to the third weight coefficient and the fourth weight coefficient to obtain the fourth color value of the third dynamic pixel point; wherein, the third weight coefficient corresponds to the first color value of the first dynamic pixel point, and the fourth weight coefficient corresponds to the second color value of the second dynamic pixel point; if the first depth value of the first dynamic pixel point is greater than the second depth value of the second dynamic pixel point, the third weight coefficient is less than the fourth weight coefficient; if the first depth value of the first dynamic pixel point is less than the second depth value of the second dynamic pixel point, the third weight coefficient is greater than the fourth weight coefficient.

[0040] In the above embodiment, when the first dynamic pixel point and the second dynamic pixel point are both pixel points of a person, the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weighted and summed according to the first weight coefficient and the second weight coefficient to obtain the fourth color value of the third dynamic pixel point. When the first dynamic pixel point or the second dynamic pixel point is not a pixel point of a person, the weight coefficients corresponding to the first color value and the second color value are determined according to the depth values of the first dynamic pixel point and the second dynamic pixel point. By distinguishing different situations and performing weighted summation on pixel points with the same coordinates to obtain the fourth color value of the third dynamic pixel point, the accuracy of the obtained color value can be improved. At the same time, through color mixing, the fourth color value can be made close to the first color value or the second color value, thereby avoiding color mutations in the animated image and improving the smoothness of the picture.

[0041] In one embodiment, determining a plurality of static pixels and a plurality of dynamic pixels in the plurality of second pixels based on the plurality of first coordinates of the plurality of first pixels and the plurality of second coordinates of the plurality of second pixels includes:

[0042] determining, according to the reprojection method, a fifth motion vector of the first coordinate of the first pixel point to be classified relative to the second coordinate of the second pixel point to be classified, where the second pixel point to be classified is any one of the plurality of second pixel points, and the first pixel point to be classified is a pixel point among the plurality of first pixel points corresponding to the second pixel point to be classified;

[0043] determining a sixth motion vector of the first pixel to be classified relative to the second pixel to be classified according to an optical flow method;

[0044] If the difference between the fifth motion vector and the sixth motion vector is less than or equal to a preset value, determining that the second pixel to be classified is a static pixel;

[0045] If the difference between the fifth motion vector and the sixth motion vector is greater than the preset value, it is determined that the second pixel to be classified is a dynamic pixel.

[0046] In the above embodiment, the motion vector calculated by the reprojection method reflects the motion vector caused by the change in camera perspective, and therefore can reflect the motion vector of a static pixel, but cannot accurately reflect the motion vector of a dynamic pixel. The motion vector calculated by the optical flow method can reflect the motion vector of a dynamic pixel. Therefore, when the motion vector calculated by the reprojection method differs significantly from the motion vector calculated by the optical flow method, it indicates that the motion vector calculated by the reprojection method is inaccurate, and further indicates that the corresponding second pixel is a dynamic pixel, thereby distinguishing between dynamic and static pixels in the second pixel.

[0047] In one embodiment, the first rendering information further includes a first view projection matrix, the second rendering information further includes a second view projection matrix, and the step of determining, according to a reprojection method, a fifth motion vector of the first coordinate of the first pixel to be classified relative to the second coordinate of the second pixel to be classified includes:

[0048] Determine the coordinates of the second pixel to be classified in the world coordinate system according to the first coordinates of the first pixel to be classified and the first view projection matrix;

[0049] determining a second coordinate of the second pixel point to be classified according to the coordinates in the world coordinate system corresponding to the second pixel point to be classified and the second view projection matrix;

[0050] The fifth motion vector is determined according to the first coordinates of the first pixel to be classified and the second coordinates of the second pixel to be classified.

[0051] In one embodiment, before determining the second coordinates of the second pixel to be classified based on the coordinates in the world coordinate system corresponding to the second pixel to be classified and the second view projection matrix, the method further includes:

[0052] When an instruction to start drawing the main scene graph corresponding to the second rendering information is detected, the second view projection matrix is obtained.

[0053] In the above embodiment, the second view projection matrix is obtained when the main scene graph is started to be drawn, and is used for subsequent determination of static pixels and dynamic pixels, which can improve data processing efficiency.

[0054] In one embodiment, the first rendering information is used to render a first image, the second rendering information is used to render a second image, the first moment is before the second moment, and the target image is used to be inserted between the first image and the second image, thereby improving the animation frame rate without increasing excessive power consumption of the electronic device.

[0055] In a second aspect, an image generating device is provided, comprising a processing module and an output module.

[0056] The processing module is used to:

[0057] Obtaining first rendering information at a first moment, the first rendering information including a plurality of first color values and a plurality of first coordinates of a plurality of first pixels, wherein the plurality of first pixels, the plurality of first color values, and the plurality of first coordinates correspond to each other in a one-to-one manner;

[0058] Obtaining second rendering information at a second moment, the second rendering information including a plurality of second color values and a plurality of second coordinates of a plurality of second pixels, the plurality of second pixels, the plurality of second color values, and the plurality of second coordinates corresponding one-to-one, and the plurality of first pixels and the plurality of second pixels corresponding one-to-one;

[0059] Determining a plurality of static pixel points and a plurality of dynamic pixel points among the plurality of second pixel points according to the plurality of first coordinates of the plurality of first pixel points and the plurality of second coordinates of the plurality of second pixel points;

[0060] determining a third color value of a third static pixel point according to the second color value of the second static pixel point, where the second static pixel point is any one of the plurality of static pixel points, and the third static pixel point is a pixel point among the plurality of third pixels points corresponding to the second static pixel point;

[0061] Determining a fourth color value of a third dynamic pixel point based on a first color value of a first dynamic pixel point and a second color value of a second dynamic pixel point, where the second dynamic pixel point is any one of the plurality of dynamic pixel points, the first dynamic pixel point is a pixel point in the plurality of first pixel points having the same coordinates as the second dynamic pixel point, and the third dynamic pixel point is a pixel point in the plurality of third pixel points having the same coordinates as the second dynamic pixel point;

[0062] An output module is used to generate a target image according to the third color value of the third static pixel point and the fourth color value of the third dynamic pixel point.

[0063] In one embodiment, the second rendering information further includes a plurality of second depth values corresponding to a plurality of static pixels, and the processing module is specifically configured to:

[0064] Acquire a depth reference map, the depth reference map including a plurality of reference pixels, the plurality of reference pixels corresponding to a plurality of reference depth values in a one-to-one manner, and the plurality of reference pixels corresponding to the plurality of third pixels in a one-to-one manner;

[0065] Obtaining a second depth value, a second color value, and a first motion vector of the second static pixel, where the first motion vector is related to the second motion vector, the second motion vector representing an offset position of a first coordinate of the first static pixel relative to a second coordinate of the second static pixel, the first static pixel being a pixel corresponding to the second static pixel among the plurality of first pixels;

[0066] determining a third coordinate of the third static pixel point according to the second coordinate of the second static pixel point and the first motion vector;

[0067] If the reference depth value corresponding to the third static pixel is equal to the second depth value of the second static pixel, the second color value of the second static pixel is used as the third color value of the third static pixel, wherein the reference depth value corresponding to the third static pixel is the reference depth value of the reference pixel having the same coordinates as the third coordinate of the third static pixel.

[0068] In one embodiment, the processing module is further configured to:

[0069] Acquire an initial depth reference map, the initial depth reference map including a plurality of initial pixel points, the plurality of initial pixel points corresponding one-to-one to a plurality of initial depth values, and the plurality of initial pixel points corresponding one-to-one to the plurality of second pixel points;

[0070] performing an update operation on the initial pixel points corresponding to the plurality of static pixel points among the plurality of initial pixel points to obtain a plurality of update values;

[0071] updating the initial depth reference map according to the multiple update values to obtain the depth reference map;

[0072] The update operation includes:

[0073] Obtaining a second depth value and a third motion vector of a fourth static pixel, where the fourth static pixel is any one of the plurality of static pixels, the third motion vector is related to the fourth motion vector, and the fourth motion vector represents an offset position of a first coordinate of a fifth static pixel relative to the second coordinate of the fourth static pixel, where the fifth static pixel is a pixel corresponding to the fourth static pixel among the plurality of first pixels;

[0074] Determining a first initial pixel point according to the second coordinate of the fourth static pixel point and the third motion vector, where the first initial pixel point is a pixel point corresponding to the fourth static pixel point among the multiple initial pixel points;

[0075] If the second depth value of the fourth static pixel is less than the initial depth value corresponding to the first initial pixel, use the second depth value of the fourth static pixel as the update value corresponding to the first initial pixel;

[0076] If the second depth value of the fourth static pixel is greater than or equal to the initial depth value corresponding to the first initial pixel, the initial depth value corresponding to the first initial pixel is used as the update value corresponding to the first initial pixel.

[0077] In one embodiment, the processing module is further configured to:

[0078] generating the second motion vector according to a difference between a first coordinate of the first static pixel and a second coordinate of the second static pixel;

[0079] A half of the second motion vector is used as the first motion vector.

[0080] In one embodiment, the processing module is specifically configured to:

[0081] Obtaining a first color value and a first identifier of the first dynamic pixel point, where the first identifier indicates whether the first dynamic pixel point is a pixel point of a person;

[0082] Obtaining a second color value and a second identifier of the second dynamic pixel point, where the second identifier indicates whether the second dynamic pixel point is a pixel point of a person;

[0083] If the first identifier represents that the first dynamic pixel point is a pixel point of a person, and the second identifier represents that the second dynamic pixel point is a pixel point of a person, the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weightedly summed according to the first weight coefficient and the second weight coefficient to obtain the fourth color value of the third dynamic pixel point; the first weight coefficient corresponds to the first color value of the first dynamic pixel point, and the second weight coefficient corresponds to the second color value of the second dynamic pixel point.

[0084] In one embodiment, the processing module is further configured to:

[0085] If the first identifier indicates that the first dynamic pixel point is not a pixel point of a person, or if the second identifier indicates that the second dynamic pixel point is not a pixel point of a person, obtaining a first depth value of the first dynamic pixel point and obtaining a second depth value of the second dynamic pixel point;

[0086] The first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weightedly summed according to the third weight coefficient and the fourth weight coefficient to obtain the fourth color value of the third dynamic pixel point; wherein, the third weight coefficient corresponds to the first color value of the first dynamic pixel point, and the fourth weight coefficient corresponds to the second color value of the second dynamic pixel point; if the first depth value of the first dynamic pixel point is greater than the second depth value of the second dynamic pixel point, the third weight coefficient is less than the fourth weight coefficient; if the first depth value of the first dynamic pixel point is less than the second depth value of the second dynamic pixel point, the third weight coefficient is greater than the fourth weight coefficient.

[0087] In one embodiment, the processing module is specifically configured to:

[0088] determining, according to the reprojection method, a fifth motion vector of the first coordinate of the first pixel point to be classified relative to the second coordinate of the second pixel point to be classified, where the second pixel point to be classified is any one of the plurality of second pixel points, and the first pixel point to be classified is a pixel point among the plurality of first pixel points corresponding to the second pixel point to be classified;

[0089] determining a sixth motion vector of the first pixel to be classified relative to the second pixel to be classified according to an optical flow method;

[0090] If the difference between the fifth motion vector and the sixth motion vector is less than or equal to a preset value, determining that the second pixel to be classified is a static pixel;

[0091] If the difference between the fifth motion vector and the sixth motion vector is greater than the preset value, it is determined that the second pixel to be classified is a dynamic pixel.

[0092] In one embodiment, the first rendering information further includes a first view projection matrix, and the second rendering information further includes a second view projection matrix. The processing module is specifically configured to:

[0093] Determine the coordinates of the second pixel to be classified in the world coordinate system according to the first coordinates of the first pixel to be classified and the first view projection matrix;

[0094] determining a second coordinate of the second pixel point to be classified according to the coordinates in the world coordinate system corresponding to the second pixel point to be classified and the second view projection matrix;

[0095] The fifth motion vector is determined according to the first coordinates of the first pixel to be classified and the second coordinates of the second pixel to be classified.

[0096] In one embodiment, the processing module is further configured to:

[0097] When an instruction to start drawing the main scene graph corresponding to the second rendering information is detected, the second view projection matrix is obtained.

[0098] In one embodiment, the first rendering information is used to render a first image, the second rendering information is used to render a second image, the first moment is before the second moment, and the target image is used to be inserted between the first image and the second image.

[0099] In a third aspect, an electronic device is provided, comprising a processor, wherein the processor is configured to execute a computer program stored in a memory to implement the image generation method as described in the first aspect above.

[0100] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image generation method as described in the first aspect is implemented.

[0101] In a fifth aspect, a chip is provided, comprising a processor coupled to a memory, wherein the processor executes a computer program or instruction stored in the memory to implement the image generation method as described in the first aspect above.

[0102] In a sixth aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes the image generation method described in the first aspect.

[0103] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 A software architecture diagram of an electronic device provided in one embodiment of the present application;

[0105] Figure 2 A schematic diagram of a target image provided in one embodiment of the present application;

[0106] Figure 3 A schematic diagram of a target image provided by another embodiment of the present application;

[0107] Figure 4 A flowchart of an image generation method provided in one embodiment of the present application;

[0108] Figure 5 A schematic diagram of the pixel correspondence relationship provided in an embodiment of the present application;

[0109] Figure 6 A schematic diagram of a coordinate mapping relationship provided in an embodiment of the present application;

[0110] Figure 7 A schematic diagram of a reprojection method provided in one embodiment of the present application;

[0111] Figure 8 A schematic diagram of the optical flow method provided in one embodiment of the present application;

[0112] Figure 9 A flowchart for determining a depth reference image provided in an embodiment of the present application;

[0113] Figure 10 A flowchart for determining a third color value of a third pixel corresponding to a plurality of static pixels provided in an embodiment of the present application;

[0114] Figure 11 A flowchart of determining a fourth color value of a third pixel corresponding to a plurality of dynamic pixels provided in an embodiment of the present application;

[0115] Figure 12 A flowchart of an implementation of an image generation method provided in one embodiment of the present application;

[0116] Figure 13 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0117] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0118] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0119] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0120] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0121] In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0122] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0123] For example, the electronic device described in the embodiments of the present application may be a mobile phone, tablet computer, handheld computer, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, media player, wearable device, or other device that can be held / operated with one hand. The embodiments of the present application do not impose any special restrictions on the specific form / type of the electronic device. The above-mentioned electronic devices include but are not limited to those equipped with Devices running Harmony OS or other operating systems.

[0124] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device.

[0125] Figure 1 It is a software structure block diagram of the electronic device according to an embodiment of the present invention.

[0126] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0127] The application layer can include a series of application packages.

[0128] like Figure 1 As shown, the application package may include games, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0129] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0130] like Figure 1 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0131] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0132] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0133] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0134] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).

[0135] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0136] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0137] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0138] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0139] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0140] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0141] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0142] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0143] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0144] A 2D graphics engine is a drawing engine for 2D drawings.

[0145] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0146] The image generation method provided in the embodiments of the present application is applied to electronic devices and is used in games or animation scenes based on 3D models.

[0147] First, some terms that appear in this article are introduced.

[0148] Rendering information, which is issued by application software (e.g., game software), includes the coordinates, color values, and depth values of each of a plurality of pixels. In embodiments of the present application, the electronic device may render the corresponding image based on the rendering information, or may determine only the color values of the pixels in the interpolated image based on the rendering information without rendering the corresponding image.

[0149] Pixels, multiple pixels make up the image displayed on the display interface of the electronic device, each pixel corresponds to a coordinate, and the coordinate of the pixel represents the position of the pixel on the display interface. For each pixel, there is a world space coordinate corresponding to the pixel in the world space. The position of the pixel on the display interface is determined according to the corresponding world space coordinate and the camera angle of view. In the embodiment of the present application, the electronic device may display an image composed of multiple pixels on the display interface, or may not display an image composed of multiple pixels.

[0150] Static pixels are pixels corresponding to static objects in the world space. Objects whose position, rotation angle, and size do not change over time in the world space are static objects.

[0151] Dynamic pixels are pixels corresponding to dynamic objects in the world space. Objects whose position, rotation angle, or size changes over time in the world space are dynamic objects.

[0152] To enhance the user's gaming experience, gaming software is developing towards high frame rates and high image quality. A high frame rate can provide users with a smoother experience, but it can also lead to high power consumption in electronic devices. Compared to generating native frame images of gaming software using the rendering information of gaming software, interpolated images generated through frame prediction methods consume less power. Therefore, generating interpolated images through frame prediction methods and then inserting them into the native frame images of gaming software, or replacing native frame images with interpolated images, can both increase the frame rate while reducing the power consumption of electronic devices.

[0153] Existing frame prediction methods generally use image warping to generate interpolated images. In some scenarios, interpolated images generated by image warping may cause distortion of objects or abnormal motion postures in the image. For example, Figure 2 As shown in FIG, for a static object 21 in a scene, object distortion may occur in the interpolated image generated by using image deformation. Figure 3 As shown, for a moving person 31 in a scene, the interpolated image generated by image deformation cannot predict the motion posture of the person 31, which will result in abnormal changes in the person's posture and poor motion smoothness.

[0154] To this end, the present application provides an image generation method, which obtains first rendering information at a first moment and second rendering information at a second moment, and determines static pixels and dynamic pixels among a plurality of second pixels based on a plurality of first coordinates of a plurality of first pixels in the first rendering information and a plurality of second coordinates of a plurality of second pixels in the second rendering information. Static pixels are pixels of static objects in an animation scene, and dynamic pixels are pixels of dynamic objects in an animation scene. Since the motion characteristics of dynamic objects and static objects are inconsistent, after determining the static pixels and the dynamic pixels, different methods are used to determine the third color value of the pixel corresponding to the static pixel in the third pixel and the fourth color value of the pixel corresponding to the dynamic pixel. This can make the third color values of the plurality of third pixels consistent with the motion characteristics of the corresponding objects in the animation scene, thereby avoiding the problem of object distortion or abnormal motion posture caused by the same frame prediction method for dynamic pixels and static pixels.

[0155] For any second static pixel among multiple static pixels, the third color value of the third pixel corresponding to the second static pixel in the target image (i.e., the interpolated image) is determined based on the color value of the second static pixel. Compared with the image deformation method in which the color values of multiple adjacent pixels are determined by the same method, the present application can avoid the object distortion problem caused by discontinuous depth of adjacent pixels. For any second dynamic pixel among multiple dynamic pixels, the third color value of the third pixel with the same coordinates as the second dynamic pixel in the target image is determined based on the color value of the second dynamic pixel and the first color value of the first dynamic pixel with the same coordinates as the second dynamic pixel. The third color value of the third pixel can be made close to the second color value of the second dynamic pixel or the first color value of the first dynamic pixel. The pixels with the same coordinates correspond to the same position on the display interface, and the third color value of the third pixel is close to the color of the first dynamic pixel or the second dynamic pixel at the same position on the display interface. The persistence effect of the human eye can be utilized. When the object moves at a fast speed, it is not easy for the corresponding position to suddenly change color, thereby improving the overall smoothness of the picture.

[0156] The image generation method provided in the embodiments of the present application is described in detail below.

[0157] like Figure 4 As shown, the frame insertion method provided in one embodiment of the present application includes the following steps.

[0158] S401: Obtain first rendering information at a first moment, where the first rendering information includes a plurality of first color values and a plurality of first coordinates of a plurality of first pixels, where the plurality of first pixels, the plurality of first color values, and the plurality of first coordinates correspond to each other in a one-to-one manner.

[0159] Specifically, the electronic device obtains the first rendering information from the application software. For example, the kernel layer of the electronic device intercepts the first rendering information sent by the application software. The first rendering information is used to render the first image. In an embodiment of the present application, after obtaining the first rendering information, the electronic device may perform subsequent steps based on the first rendering information, and simultaneously render the first image based on the first rendering information and display the first image on the display interface, or may perform subsequent steps based only on the first rendering information.

[0160] A plurality of first pixel points constitute a first image, the first coordinates represent positions of the first pixel points on the display interface of the electronic device, and the first color value may be an RGB value or a pixel value.

[0161] S402: Obtain second rendering information at a second moment, where the second rendering information includes a plurality of second color values and a plurality of second coordinates for a plurality of second pixels, the plurality of second pixels, the plurality of second color values, and the plurality of second coordinates corresponding one-to-one, and the plurality of first pixels and the plurality of second pixels corresponding one-to-one.

[0162] Among them, the first moment is before the second moment, and the time interval between the first moment and the second moment is one or more rendering cycles. The electronic device obtains the second rendering information from the application software, and the second rendering information is used to render the second image. In an embodiment of the present application, the electronic device may perform subsequent steps according to the second rendering information after obtaining the second rendering information, and at the same time render the second image according to the second rendering information and display the second image on the display interface, or may perform subsequent steps only according to the second rendering information.

[0163] A plurality of second pixel points constitute a second image, the second coordinates represent positions of the second pixel points on the display interface of the electronic device, and the second color value may be an RGB value or a pixel value.

[0164] For any first pixel point, the correspondence between the first pixel point and the second pixel point means that the first pixel point and the second pixel point correspond to the same world space coordinates. For example, after determining the world space coordinate system, different images can be obtained according to different camera perspectives. Figure 5 As shown, first image 51 and second image 52 are images taken from different camera perspectives. First image 51 includes multiple first pixels, and second image 52 includes multiple second pixels. First pixel A and second pixel B correspond to the same point on a static object, i.e., to the same world space coordinates. The first coordinate of first pixel A and the second coordinate of second pixel B are located in the same coordinate system. The first coordinate of first pixel A in the first image and the second coordinate of second pixel B are different in the second image. The pixel corresponding to first pixel A is second pixel B.

[0165] S403: Determine a plurality of static pixel points and a plurality of dynamic pixel points in the plurality of second pixel points according to the plurality of first coordinates of the plurality of first pixel points and the plurality of second coordinates of the plurality of second pixel points.

[0166] Specifically, for each first pixel point, it is determined whether the second static pixel point is a static pixel point or a dynamic pixel point according to the positional relationship between the first coordinate of the first pixel point and the second coordinate of the second pixel point corresponding to the first pixel point.

[0167] In one embodiment, taking the second pixel to be classified among the second pixel points as an example, the first pixel to be classified is the pixel point among the first pixel points corresponding to the second pixel to be classified. The fifth motion vector of the first coordinate of the first pixel to be classified relative to the second coordinate of the second pixel to be classified is determined according to the reprojection method, and the sixth motion vector of the first coordinate of the first pixel to be classified relative to the second coordinate of the second pixel to be classified is determined according to the optical flow method. If the difference between the fifth motion vector and the sixth motion vector is less than or equal to the preset value, the second pixel to be classified is determined to be a static pixel point. If the difference between the fifth motion vector and the sixth motion vector is greater than the preset value, the second pixel to be classified is determined to be a dynamic pixel point. The preset value can be set according to actual needs, for example, the preset value is the length of two pixels.

[0168] For the second pixel point to be classified, the motion vector calculated by the reprojection method reflects the motion vector of the pixel point caused by the change of the camera. If the pixel point is a pixel point of a static object, the motion vector calculated by the reprojection method is the actual motion vector of the pixel point. If the pixel point is a pixel point of a dynamic object, the motion vector calculated by the reprojection method cannot represent the actual motion vector of the pixel point. The motion vector calculated by the optical flow method can accurately reflect the motion vector of the pixel point of a dynamic object. For the same pixel point, if the motion vector calculated by the reprojection method is significantly different from the motion vector calculated by the optical flow method, it means that the motion vector calculated by the reprojection method is inaccurate, and the pixel point is changed to a dynamic pixel point. Therefore, by comparing the fifth motion vector calculated by the reprojection method and the sixth motion vector calculated by the optical flow method, it can be determined whether the second pixel point to be classified is a static pixel point or a dynamic pixel point.

[0169] In one embodiment, the electronic device uses a reprojection method to determine a fifth motion vector of a first coordinate of a first pixel to be classified relative to a second coordinate of a second pixel to be classified as follows.

[0170] First, the coordinate systems used in the reprojection method and the mapping relationship between coordinate systems are introduced.

[0171] The image displayed on the display interface is a mapping of objects in world space. For every pixel in the image, there is a corresponding point in the world space coordinate system. By using the mapping relationship between coordinate systems, the world space coordinates corresponding to any pixel in the world space coordinate system can be determined.

[0172] The coordinates of a pixel point reflect its position on the display interface, and the coordinate system in which the pixel point coordinates are located is the device space coordinate system. When rendering an image, in addition to the device space coordinate system and the world space coordinate system, mapping relationships between the local space coordinate system, the observation space coordinate system, the homogeneous clipping space coordinate system, the normalized device space coordinate system, and the viewport space coordinate system are also required.

[0173] Among them, the local space coordinate system is a coordinate system established based on each object. The coordinates of the local space coordinate system represent the position of a point on the object. The world space coordinate system is also the world coordinate system. The coordinates of any point in the world coordinate system remain unchanged. The observation space coordinate system is the coordinate system of the camera's perspective and changes with the change of the camera's pose. The homogeneous clipping space represents the area that can be displayed when the camera is framing. The coordinates of the homogeneous clipping space coordinate system represent the coordinates within this displayable area. The normalized device space coordinate system represents the coordinate system after the device space coordinate system is normalized. In the normalized device space coordinate system, the coordinates of each point are in the range of 0 to 1 or -1 to 1. The viewport space refers to the space that maps each coordinate in the device space to the range of (0,0) to (1,1).

[0174] For example, the mapping relationship between the coordinate systems is as follows: Figure 6 As shown. For any coordinate in the local space coordinate system, the coordinate in the world space coordinate system can be obtained from the simulation matrix. For any coordinate in the world space coordinate system, the coordinate in the local space coordinate system can be obtained from the inverse model matrix. For any coordinate in the world space coordinate system, the coordinate in the view space coordinate system can be obtained from the view matrix. For any coordinate in the view space coordinate system, the coordinate in the world space coordinate system can be obtained from the inverse view projection matrix. For any coordinate in the view space coordinate system, the coordinate in the homogeneous clip space coordinate system can be obtained from the projection matrix. For any coordinate in the homogeneous clip space coordinate system, the coordinate in the view space coordinate system can be obtained from the inverse projection matrix. For any coordinate in the world space coordinate system, the coordinate in the homogeneous clip space coordinate system can be obtained from the view projection matrix. For any coordinate in the homogeneous clip space coordinate system, the coordinate in the world space coordinate system can be obtained from the inverse view projection matrix. For any coordinate in the homogeneous clip space coordinate system, the coordinate in the normalized device space coordinate system can be obtained through homogeneous division. For any coordinate in the normalized device space coordinate system, the coordinate in the viewport space coordinate system can be obtained through the mapping matrix.

[0175] The mapping relationship between the above coordinate systems can be obtained from the rendering information.

[0176] In one embodiment, the first rendering information further includes a first view projection matrix, and the second rendering information further includes a second view projection matrix. The first view projection matrix is an inverse view projection matrix, and the second view projection matrix is a view projection matrix. A method for determining a fifth motion vector of the first coordinate of the first pixel to be classified relative to the second coordinate of the second pixel to be classified using a reprojection method is as follows.

[0177] The electronic device first determines the coordinates of the second pixel point to be classified in the world coordinate system according to the first coordinates of the first pixel point to be classified and the first view projection matrix.

[0178] Specifically, the coordinates of the normalized device space corresponding to the first pixel to be classified, pre_ndc, are constructed according to the formula pre_ndc=(u*2.0–1.0, v*2.0–1.0, depth*2.0–1.0, 1.0). The coordinates of the normalized device space corresponding to the first pixel to be classified are four-dimensional vectors, u represents the horizontal coordinate of the first coordinate of the first pixel to be classified, v represents the vertical coordinate of the first coordinate of the first pixel to be classified, and depth represents the depth value of the first pixel to be classified. Then, according to the formula pre_world=VP t-1 -1 *pre_ndc, determine the world space coordinates pre_world corresponding to the first pixel to be classified. t-1 -1 Represents the first view projection matrix. The first pixel to be classified and the second pixel to be classified correspond to the same point in the world space. The coordinates of the first pixel to be classified in the world space are the coordinates of the second pixel to be classified in the world space.

[0179] Afterwards, the electronic device determines the second coordinates of the second pixel point to be classified according to the coordinates in the world coordinate system (ie, the view space coordinate system) corresponding to the second pixel point to be classified and the second view projection matrix.

[0180] Specifically, according to the formula cur_clip=VP t *pre_world determines the coordinates cur_clip of the clipping space corresponding to the second pixel to be classified, where VP t Represents the second view projection matrix. Then, according to the formulas cur_ndc.x = cur_clip.x / cur_clip.w and cur_ndc.y = cur_clip.y / cur_clip.w, the coordinates of the device space corresponding to the second pixel to be classified (cur_ndc.x, cur_ndc.y) are determined. The coordinates of the device space are two-dimensional vectors. The coordinates of the device space are the second coordinates of the second pixel to be classified. cur_ndc.x and cur_ndc.y are the horizontal and vertical coordinates of the second coordinates respectively. Among them, cur_clip.x represents the first element in the coordinates of the clipping space corresponding to the second pixel to be classified, cur_clip.y represents the second element in the coordinates of the clipping space corresponding to the second pixel to be classified, and cur_clip.w represents the fourth element in the coordinates of the clipping space corresponding to the second pixel to be classified.

[0181] Afterwards, the electronic device determines the fifth motion vector according to the first coordinates of the first pixel point to be classified and the second coordinates of the second pixel point to be classified.

[0182] Specifically, the fifth motion vector (mv.x, mv.y) is determined according to the formulas mv.x = u - (cur_ndc.x * 0.5 + 0.5) and mv.y = v - (cur_ndc.y * 0.5 + 0.5). The fifth motion vector is represented by coordinates, with mv.x and mv.y representing the abscissa (i.e., horizontal displacement) and ordinate (i.e., vertical displacement) of the fifth motion vector, respectively.

[0183] The optical flow method is an image processing method used to determine the corresponding pixel point between the first pixel point and the second pixel point by using the brightness changes of each pixel in a first image composed of the first pixel points and a second image composed of the second pixel points, thereby determining the sixth motion vector of the first pixel point to be classified relative to the second pixel point to be classified. Determining the motion vector of the corresponding pixel point using the optical flow method can adopt matching-based methods, gradient-based methods, frequency-domain-based methods, phase-based methods, and neural dynamics methods. These methods are existing technologies and are not further described here.

[0184] Based on the fifth motion vector and the sixth motion vector, it can be determined whether the second pixel to be classified is a static pixel or a dynamic pixel. The above method is used to determine whether all pixels in the second pixel set are static pixels or dynamic pixels, thereby determining multiple static pixels and multiple dynamic pixels in the second pixel set.

[0185] For example, Figure 7 As shown, the image composed of the first pixel points is the first image 71, and the image composed of the second pixel points is the second image 72. For each pixel point in the first image, the reprojection method is used to determine the coordinates of the corresponding pixel point in the second image, and then the motion vector corresponding to each pixel point can be determined.

[0186] like Figure 8 As shown, the image composed of the first pixel points is the first image 81, and the image composed of the second pixel points is the second image 82. According to the optical flow method, the coordinates of each pixel point in the first image in the second image can be determined, and then the motion vector corresponding to each pixel point can be determined.

[0187] Thus, each second pixel corresponds to two motion vectors. If the difference between the two motion vectors is less than or equal to a preset value, the second pixel is determined to be a static pixel. If the difference between the two motion vectors is greater than the preset value, the second pixel is determined to be a dynamic pixel.

[0188] After determining the static pixels and dynamic pixels in the second pixel point, the electronic device can set different identifiers for the static pixels and dynamic pixels respectively, so as to subsequently determine the color values of the corresponding pixels in the interpolated image based on the static pixels and dynamic pixels respectively.

[0189] S404: Determine the third color value of the third static pixel point based on the second color value of the second static pixel point, where the second static pixel point is any one of the multiple static pixel points, and the third static pixel point is a pixel point among the multiple third pixel points corresponding to the second static pixel point.

[0190] Specifically, a plurality of third pixel points constitute a target image. Taking a second static pixel point among a plurality of static pixel points as an example, there is a first static pixel point corresponding to the second static pixel point among a plurality of first pixel points, and the coordinates of the world space coordinate system corresponding to the second static pixel point and the first static pixel point are the same. Similarly, there is a third static pixel point corresponding to the second static pixel point among a plurality of third pixel points, and the coordinates of the world space coordinate system corresponding to the third static pixel point and the second static pixel point are the same. The second motion vector is used to represent the offset position of the first coordinate of the first static pixel point relative to the second coordinate of the second static pixel point. According to the time interval between the target image and the first image or the second image, the first motion vector can be obtained. The first motion vector represents the offset position of the third coordinate of the third static pixel point relative to the second coordinate of the second static pixel point. For example, the target image is used to be inserted between the first image and the second image. According to a fixed frame rate, the first motion vector is half of the second motion vector.

[0191] The calculation method of the second motion vector is the same as the calculation method of the fifth motion vector, that is, the second motion vector is the difference between the first coordinate of the first static pixel and the second coordinate of the second static pixel. For example, the first coordinate of the first static pixel is (x1, y1), and the second coordinate of the second static pixel is (x2, y2), then the second motion vector is (x1-x2, y1-y2). Half of the second motion vector can be 1 / 2 of the second motion vector or close to 1 / 2. For example, half of the second motion vector is (1 / 2(x1-x2), 1 / 2(y1-y2)).

[0192] The third coordinate of the third static pixel can be obtained based on the second coordinate of the second static pixel and the first motion vector. Therefore, the third static pixel is obtained by shifting the second static pixel, and the third color value of the third static pixel can be determined based on the second color value of the second static pixel and the third coordinate of the third static pixel.

[0193] The above method is used for each static pixel point in the multiple static pixel points to obtain a third pixel point corresponding to each static pixel point, and the multiple third pixel points constitute a target image.

[0194] Each third pixel is obtained by offsetting the static pixel, and the offset position is represented by the corresponding motion vector. Since the motion vectors corresponding to the static pixels are different, different static pixels may be offset to the same third pixel in the target image according to the motion vectors corresponding to the static pixels. In the Unity engine (a game development platform), the depth value of a pixel represents the distance between the pixel and the screen of the electronic device. The closer the distance to the screen, the smaller the depth value of the pixel, and the farther the distance from the screen, the larger the depth value of the pixel. The depth values of different static pixels offset to the same third pixel of the target image may also be different. The static pixels close to the screen will block the static pixels far from the screen. In order to ensure that there is a correct occlusion relationship when different static pixels are offset to the same third pixel of the target image, it is necessary to establish a depth reference map based on the depth values of each static pixel, and use the depth reference map to guide the third color value of the third static pixel corresponding to each static pixel.

[0195] In one embodiment, the specific process of generating a depth reference map is as follows.

[0196] The electronic device obtains an initial depth reference map, which includes a plurality of initial pixels, a one-to-one correspondence between the plurality of initial pixels and a plurality of initial depth values, and a one-to-one correspondence between the plurality of initial pixels and a plurality of second pixels. The initial depth value is a preset maximum depth value.

[0197] An update operation is performed on the initial pixel points corresponding to the plurality of static pixel points among the plurality of initial pixel points to obtain a plurality of update values, each update value corresponding to an initial pixel point, and an initial depth reference map is updated according to the update values to obtain a depth reference map.

[0198] Taking a fourth static pixel among the static pixels as an example, the process of performing an update operation on a first initial pixel among the initial pixels corresponding to the fourth static pixel is as follows.

[0199] The electronic device obtains a second depth value and a third motion vector of a fourth static pixel. The second depth value is obtained from the second rendering information. The third motion vector represents an offset position of a pixel corresponding to the fourth static pixel among a plurality of third pixels relative to the second coordinate of the fourth static pixel. The third motion vector is related to the fourth motion vector, and the fourth motion vector represents an offset position of the first coordinate of a fifth static pixel relative to the second coordinate of the fourth static pixel. The fifth static pixel is a pixel corresponding to the fourth static pixel among a plurality of first pixels.

[0200] The calculation method of the fourth motion vector is the same as the calculation method of the fifth motion vector, and the third motion vector may be half of the fourth motion vector.

[0201] The electronic device then determines the coordinates of the first initial pixel based on the second coordinates of the fourth static pixel and the third motion vector. The coordinates of the first initial pixel are the coordinates of the fourth static pixel in the target image after being shifted by the third motion vector, i.e., the coordinates of the corresponding third pixel in the target image.

[0202] If the second depth value of the fourth pixel is less than the initial depth value of the first initial pixel, the second depth value of the fourth static pixel is used as the updated value corresponding to the first initial pixel. If the second depth value of the fourth static pixel is greater than or equal to the initial depth value corresponding to the first initial pixel, the initial depth value of the first initial pixel is not updated, that is, the initial depth value corresponding to the first initial pixel is used as the updated value corresponding to the first initial pixel.

[0203] For each static pixel point, the above method is used to perform an update operation on the initial pixel point corresponding to the static pixel point. After all the initial pixel points corresponding to the static pixel point are updated, the updated initial pixel point is the reference pixel point, and the update value corresponding to the updated initial pixel point is the reference depth value, that is, the updated initial depth reference map is the depth reference map.

[0204] In one embodiment, the specific process of the electronic device generating the depth reference map is as follows: Figure 9 shown.

[0205] After determining the static pixels and the dynamic pixels from the plurality of second pixels, the electronic device sets the static pixels and the dynamic pixels to different dynamic identifiers, respectively. For example, the dynamic identifier of the static pixels is set to a first preset value (e.g., 0), and the dynamic identifier of the dynamic pixels is set to a second preset value (e.g., 1). Based on the identifiers of each pixel from the plurality of second pixels, the electronic device can obtain a dynamic object mask map, which indicates the position of the dynamic object in the second image.

[0206] The electronic device establishes an initial depth reference map, and each initial pixel point in the initial depth reference map corresponds to an initial depth value. The electronic device samples multiple second pixel points according to the dynamic object mask map to obtain a dynamic identifier corresponding to each second pixel point. For each second pixel point, it is determined whether the dynamic identifier is a second preset value. If the dynamic identifier is the second preset value, it means that the second pixel point is a dynamic pixel point, and the initial depth value of the initial pixel point with the same coordinates as the second pixel point in the initial depth reference map is updated to a preset minimum depth value (for example, 0). If the identifier of the second pixel point is not the second preset value, it is determined that the second pixel point is a static pixel point, and an update operation is performed.

[0207] Specifically, a motion vector map and a depth reference map of the second image are obtained, wherein the motion vector map includes motion vectors corresponding to a plurality of second pixels, and the depth reference map includes depth values corresponding to a plurality of second pixels. The motion vector map and the depth reference map are sampled to obtain a motion vector and a depth value corresponding to each second pixel.

[0208] For a fourth static pixel among the static pixels, the motion vector is the third motion vector, and the depth value is the second depth value. The coordinates of the third static pixel after the second coordinate is offset are determined based on the third motion vector. The offset coordinates are the coordinates of the first initial pixel among the initial pixels corresponding to the fourth static pixel. If the second depth value of the fourth static pixel is less than the initial depth value corresponding to the first initial pixel, the second depth value of the fourth static pixel is used as the update value corresponding to the first initial pixel. Otherwise, the update operation is not performed.

[0209] Each static pixel is traversed, and the initial pixel corresponding to the static pixel is updated using the above method to obtain an updated depth reference map. It can be understood that after the initial depth value of any pixel in the initial depth reference map is updated, a new initial depth reference map is obtained, and the new initial depth reference map is used as the initial depth reference map for the next update operation. The depth reference map is obtained after repeated iterations and traversing each pixel in the current frame image.

[0210] Since the initial depth value of the pixel point identified as the second preset value (i.e., the dynamic pixel point) is replaced with the minimum depth value, when updating the initial depth value of each pixel point in the initial depth reference map, there will not be a situation where the depth value of the second pixel point is less than the initial depth value of the corresponding initial pixel point, and thus the initial depth value of the initial pixel point corresponding to the dynamic pixel point in the initial depth reference map will not be updated. Therefore, when generating the depth reference map, the initial depth value of the initial pixel point corresponding to the dynamic pixel point is not replaced, and then when the color value corresponding to the static pixel point is subsequently determined, the color value of the third pixel point corresponding to the dynamic pixel point is not affected.

[0211] The depth reference map is obtained by updating the initial depth value in the initial depth reference map. The updated reference depth value is the minimum depth value of the offset position in the target image after each static pixel is offset. It is the depth value of the unobstructed pixel at the offset position in the target image, that is, the reference depth value of each pixel in the depth reference map is the depth value of the pixel presented on the display interface of the electronic device after offset, which represents the depth value of each pixel in the target image. According to the reference depth value corresponding to each third pixel, the static pixel corresponding to the third pixel can be determined, and then the color of the static pixel corresponding to the third pixel can be determined, and then the third color of the third pixel corresponding to the static pixel can be determined.

[0212] In one embodiment, a specific process of determining the third color value of the third static pixel according to the second color value of the second static pixel is as follows.

[0213] The electronic device first obtains a depth reference map, which includes multiple reference pixels, each of which corresponds to a plurality of reference depth values, and each of which corresponds to a plurality of third pixels. The electronic device determines a first motion vector based on the second motion vector, and then determines a third coordinate of a third static pixel based on the second coordinate of the second static pixel and the first motion vector.

[0214] The electronic device may determine, based on the third coordinate of the third static pixel, a reference pixel having the same coordinates as the third coordinate of the third static pixel, and use the reference depth value corresponding to the reference pixel as the reference depth value corresponding to the third static pixel. Simultaneously, the electronic device obtains the second depth value of the second static pixel from the second rendering information.

[0215] If the second depth value of the second static pixel is equal to the reference depth value corresponding to the third static pixel, it means that the third static pixel is obtained by offsetting the second static pixel, and the second color value of the second static pixel is used as the third color value of the third static pixel. If the second depth value of the second static pixel and the reference depth value corresponding to the third static pixel are not equal, the third color value of the third pixel is kept blank.

[0216] The electronic device traverses all static pixel points, and for each static pixel point, uses the above method to determine the third color value of the third pixel point corresponding to the static pixel point.

[0217] In one embodiment, the specific process of determining the third color value of the third pixel corresponding to the plurality of static pixels is as follows: Figure 10 shown.

[0218] The electronic device obtains a depth reference map, samples the depth reference map, and obtains a reference depth value for each reference pixel. For each reference pixel, determine whether the reference depth value is a minimum depth value. If the reference depth value of the reference pixel is the minimum depth value, set the color value of the third pixel with the same coordinates as the reference pixel among the plurality of third pixels as a specific identifier (e.g., fiag1) for subsequent processing. If the reference depth value of the reference pixel is not the minimum depth value, determine the third color value of the third pixel with the same coordinates as the reference pixel using the following method.

[0219] Specifically, a color map, a motion vector map and a depth map of the second image are obtained. The color map is sampled to obtain the second color value of each second pixel. The motion vector map is sampled to obtain the motion vector of each second pixel. The depth map is sampled to obtain the second depth value of each second pixel. For one of the second static pixel points among the static pixel points, the coordinates of the third static pixel point corresponding to the second static pixel point are determined according to the first motion vector of the second static pixel point. Afterwards, the electronic device determines the reference depth value of the reference pixel point with the same coordinates as the third static pixel point in the depth reference map. If the reference depth value of the reference pixel point with the same coordinates as the third static pixel point is equal to the second depth value of the second static pixel point, the second color value of the second static pixel point is used as the third color value of the third static pixel point. Traverse all the static pixels and use the above method to determine the third color value of the third pixel point corresponding to the static pixel point.

[0220] By determining the third color value of the third pixel corresponding to the static pixel pixel by pixel, the problem of distortion in the target image when the depth values of adjacent pixels are discontinuous can be solved, thereby improving the image quality of the obtained target image.

[0221] S405: Determine the fourth color value of the third dynamic pixel point based on the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point, where the second dynamic pixel point is any one of the multiple dynamic pixel points, the first dynamic pixel point is a pixel point among the multiple first pixel points with the same coordinates as the second dynamic pixel point, and the third dynamic pixel point is a pixel point among the multiple third pixel points with the same coordinates as the second dynamic pixel point.

[0222] Specifically, taking one of the second dynamic pixel points among the multiple dynamic pixel points as an example, there is a first dynamic pixel point with the same coordinates as the second dynamic pixel point among the multiple first pixel points, and there is a third dynamic pixel point with the same coordinates as the second dynamic pixel point among the multiple third pixel points. The first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are mixed to obtain the fourth color value of the third dynamic pixel point. The color mixing method is determined based on the first identifier and the first depth value of the first dynamic pixel point and the second identifier and the second depth value of the second pixel point. Among them, the first identifier represents whether the first dynamic pixel point is a pixel point of a person, and the second identifier represents whether the second dynamic pixel point is a pixel point of a person. For example, the first identifier can be 0 or 1, 1 is a person identifier, indicating that the first dynamic pixel point is a pixel point of a person, and 0 is a non-person identifier, indicating that the first dynamic pixel point is not a pixel point of a person.

[0223] In one embodiment, a method for determining the fourth color value of the third dynamic pixel is as follows.

[0224] The electronic device obtains a first color value and a first identifier of a first dynamic pixel point, and obtains a second color value and a second identifier of a second dynamic pixel point. If the first identifier indicates that the first dynamic pixel point is a pixel point of a person, and the second identifier indicates that the second dynamic pixel point is a pixel point of a person, it means that the first dynamic pixel point and the second dynamic pixel point are both pixel points of a person. Since the coordinates of the first dynamic pixel point and the second dynamic pixel point are the same, it means that on the display interface, the first dynamic pixel point and the second dynamic pixel point are located at positions where the person overlaps. Then, the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weighted and summed according to a first weight coefficient and a second weight coefficient to obtain a fourth color value of the third dynamic pixel point; the first weight coefficient corresponds to the first color value of the first dynamic pixel point, and the second weight coefficient corresponds to the second color value of the second dynamic pixel point. When the first dynamic pixel point and the second dynamic pixel point are located at positions where the person overlaps, the difference between the first weight coefficient and the second weight coefficient is small. For example, the first weight coefficient is between 0.45 and 0.55, and the sum of the first weight coefficient and the second weight coefficient is 1.

[0225] Exemplarily, the color value is an RGB value, and the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weightedly summed according to the first weight coefficient and the second weight coefficient. This may refer to weighted summing of the R value in the first color value and the R value in the second color value according to the first weight coefficient and the second weight coefficient, weighted summing of the G value in the first color value and the G value in the second color value according to the first weight coefficient and the second weight coefficient, weighted summing of the B value in the first color value and the B value in the second color value according to the first weight coefficient and the second weight coefficient, and obtaining the fourth color value of the third dynamic pixel point according to the R, G, and B values obtained after the weighted sum.

[0226] If the first identifier indicates that the first dynamic pixel is not a pixel of a person, or the second identifier indicates that the second dynamic pixel is not a pixel of a person, indicating that the first dynamic pixel and the second dynamic pixel do not overlap, the electronic device obtains the first depth value of the first dynamic pixel and the second depth value of the second dynamic pixel, and performs a weighted summation of the first color value of the first dynamic pixel and the second color value of the second dynamic pixel according to the third weight coefficient and the fourth weight coefficient to obtain a fourth color value of the third dynamic pixel. The third weight coefficient corresponds to the first color value of the first dynamic pixel, and the fourth weight coefficient corresponds to the second color value of the second dynamic pixel. If the first depth value of the first dynamic pixel is greater than the second depth value of the second dynamic pixel, the third weight coefficient is less than the fourth weight coefficient. If the first depth value of the first dynamic pixel is less than the second depth value of the second dynamic pixel, the third weight coefficient is greater than the fourth weight coefficient. That is, the color value of a pixel with a smaller depth value corresponds to a larger weight coefficient, and the color value of a pixel with a larger depth value corresponds to a smaller weight coefficient.

[0227] Pixels with smaller depth values are closer to the screen, while pixels with larger depth values are farther from the screen. Pixels with smaller depth values will obscure pixels with larger depth values. By setting the color values of pixels with smaller depth values to larger weight coefficients and the color values of pixels with larger depth values to smaller weight coefficients, the resulting mixed fourth color value can conform to actual image display rules. For example, the sum of the third and fourth weight coefficients is 1, and the weight coefficient corresponding to the color value of pixels with smaller depth values is between 0.6 and 0.8.

[0228] All dynamic pixel points are traversed, and the third pixel point corresponding to each dynamic pixel point uses the above method to obtain the corresponding color value.

[0229] In one embodiment, the specific process of determining the color value of the third pixel corresponding to the plurality of dynamic pixel points is as follows: Figure 11 shown.

[0230] Obtain a depth template map and a color map of the second image and a depth template map and a color map of the first image. The depth template map includes a plurality of pixels, each pixel corresponding to a depth value and an identification value. The depth template map includes a plurality of pixels corresponding to a second pixel one-to-one, and the identification value is used to characterize whether the corresponding second pixel is a pixel of a person. Sample the depth template map of the second image to obtain second depth values and identification values corresponding to a plurality of dynamic pixel points. Sample the depth template map of the first image to obtain a first depth value and identification value of a first pixel with the same coordinates as the plurality of dynamic pixel points. Sample the color map of the second image to obtain a second color value corresponding to the plurality of dynamic pixel points. Sample the color map of the first image to obtain a first color value of a first pixel with the same coordinates as the plurality of dynamic pixel points.

[0231] For a second dynamic pixel point among the plurality of dynamic pixel points, the identification value of the second dynamic pixel point is the second identification, and the identification value of a first dynamic pixel point among the plurality of first pixel points having the same coordinates as the second dynamic pixel point is the first identification. If the first identification indicates that the first dynamic pixel point is a pixel point of a person, and the second identification indicates that the second dynamic pixel point is a pixel point of a person, a weighted sum is performed on the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point based on a first weight coefficient corresponding to the first color value and a second weight coefficient corresponding to the second color value to obtain a fourth color value of the third dynamic pixel point.

[0232] If the first identifier indicates that the first dynamic pixel point is not a pixel point of a person, or the second identifier indicates that the second dynamic pixel point is not a pixel point of a person, the electronic device obtains the first depth value of the first dynamic pixel point and the second depth value of the second dynamic pixel point. If the first depth value of the first dynamic pixel point is greater than the second depth value of the second dynamic pixel point, the third weight coefficient is less than the fourth weight coefficient; if the first depth value of the first dynamic pixel point is less than the second depth value of the second dynamic pixel point, the third weight coefficient is greater than the fourth weight coefficient. After determining the third weight coefficient and the fourth weight coefficient, the electronic device performs a weighted summation of the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point according to the third weight coefficient corresponding to the first color value and the fourth weight coefficient corresponding to the second color value to obtain the fourth color value of the third dynamic pixel point.

[0233] Since the dynamic object is in motion, the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are mixed to obtain the fourth color value of the third dynamic pixel point, so that the fourth color value of the third dynamic pixel point is close to the first color value of the first dynamic pixel point or the second color value of the second dynamic pixel point. Since the coordinates of the first dynamic pixel point, the second dynamic pixel point, and the third dynamic pixel point are the same, that is, the positions on the display interface are the same, the persistence effect of the human eye can be utilized. When the electronic device displays the target image, the visual effect will not be affected when there are differences in color, thereby improving the smoothness of the picture.

[0234] S406: Generate a target image according to the third color value of the third static pixel and the fourth color value of the third dynamic pixel.

[0235] Specifically, the third static pixel is any one of the plurality of third pixels corresponding to the plurality of second static pixels, and the third dynamic pixel is any one of the plurality of third pixels corresponding to the plurality of second dynamic pixels. The above method is used to obtain color values for both the third pixel corresponding to the plurality of second static pixels and the third pixel corresponding to the plurality of second dynamic pixels. The electronic device generates a target image based on the color values of the plurality of third pixels and the corresponding third coordinates.

[0236] In one embodiment, after obtaining the color values corresponding to multiple static pixel points and multiple dynamic pixel points, if there are other third pixel points with undetermined color values in the target image, the color values of the other third pixel points can be determined by a color filling method. For example, for any third pixel point, if the color value of the third pixel point is not obtained by the aforementioned method, the color value of the adjacent third pixel point can be used as the color value of the third pixel point, thereby obtaining a complete target image.

[0237] In one embodiment, after obtaining the target image, the electronic device sends the target image to a cache, reads the target image from the cache, and displays it on a display interface.

[0238] In one embodiment, the electronic device can render a first image according to the first rendering information, display the first image on the display interface, and then display the target image on the display interface, and then render a second image according to the second rendering information, and display the second image on the display interface, so that the target image is inserted between the first image and the second image, thereby improving the frame rate of the animation while reducing power consumption, thereby improving the smoothness of the animation.

[0239] In another embodiment, after the electronic device obtains the target image according to the first rendering information and the second rendering information, the target image is used to replace the first image and the second image, thereby reducing the power consumption of the electronic device according to the user's frame rate requirements.

[0240] In the above embodiment, by dividing the second pixel points into dynamic pixel points of dynamic objects and static pixel points of static objects, and determining the third color value of the third pixel point corresponding to the multiple static pixel points in the target image pixel by pixel based on the pixel values of the multiple static pixel points, the problem of image distortion caused by discontinuous depth values of the static object can be avoided. By mixing the color value of the dynamic pixel point and the first pixel point with the same coordinates as the dynamic pixel point, the fourth color value of the third pixel point with the same coordinates as the dynamic pixel point is obtained, so that the third pixel points in the target image will not have a sudden change in color relative to the pixel points in the adjacent frame, thereby utilizing the persistence effect of the human eye so that the target image and the pixel points in the adjacent frame image will not affect the visual effect of the dynamic object even if there is a difference in color, thus avoiding the distortion during the movement of the object, and thus improving the smoothness of the picture.

[0241] The process of the image generation method provided in one embodiment of the present application is as follows: Figure 12 shown.

[0242] When the electronic device detects the instruction issued by the game software to start drawing the game frame, it determines whether to start drawing the main scene graph of the second image (the image corresponding to the second rendering information). When the instruction to start drawing the main scene graph of the second image is detected, the second view projection matrix is intercepted from the second rendering information issued by the game software. The second view projection matrix is the view projection matrix of the second image, for example, it can be a view projection matrix that converts the coordinates of the world space into the coordinates of the homogeneous clipping space. The second image includes a main scene graph and a user interface (UI) graph. The UI graph refers to an image used to realize human-computer interaction, and the main scene graph is an image other than the UI graph. The electronic device first draws the main scene graph and then draws the UI graph. The main scene graph includes the second coordinates and the second color value of each second pixel point in the main scene graph.

[0243] After obtaining the second view projection matrix, the electronic device determines whether the main scene graph is finished being drawn. After the main scene graph is finished being drawn, the electronic device intercepts the depth map of the second image from the second rendering information, and then intercepts the main scene graph.

[0244] After the second image is rendered, the electronic device calculates the motion vector corresponding to each second pixel using a reprojection method based on the first rendering information, the second view projection matrix, the depth map, and the main scene map. Then, based on the main scene map of the first image and the main scene map of the second image corresponding to the first rendering information, the electronic device calculates the motion vector corresponding to each second pixel using an optical flow method. For each second pixel, the electronic device determines whether the second pixel is a dynamic pixel or a static pixel by comparing the motion vectors calculated using the two methods.

[0245] After determining the dynamic pixel points and the static pixel points, the third color value of the third pixel point corresponding to the multiple static pixel points is first determined by the pixel-by-pixel movement method, and then the fourth color value of the third pixel point with the same coordinates as the multiple dynamic pixel points is determined by the color mixing method. Finally, the target image is generated according to the third color values or fourth color values corresponding to the multiple third pixel points.

[0246] In the above embodiment, obtaining the second view projection matrix at the beginning of drawing the main scene graph can save data reception time and improve the efficiency of generating the target image. After the main scene graph is drawn, the main scene graph is intercepted and the third color value of the third pixel in the target image is determined based on the second color value carried in the main scene graph. This can improve the efficiency of generating the target image without affecting the display effect of the target image.

[0247] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0248] For example, Figure 13 A structural diagram of the electronic device 100 is shown.

[0249] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0250] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0251] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0252] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0253] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0254] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0255] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0256] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0257] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0258] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0259] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0260] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0261] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0262] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0263] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0264] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0265] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0266] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0267] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0268] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0269] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0270] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0271] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image generation method, characterized in that: include: Obtaining first rendering information at a first moment, the first rendering information including a plurality of first color values and a plurality of first coordinates of a plurality of first pixels, wherein the plurality of first pixels, the plurality of first color values, and the plurality of first coordinates correspond to each other in a one-to-one manner; Obtaining second rendering information at a second moment, the second rendering information including a plurality of second color values and a plurality of second coordinates of a plurality of second pixels, the plurality of second pixels, the plurality of second color values, and the plurality of second coordinates corresponding one-to-one, and the plurality of first pixels and the plurality of second pixels corresponding one-to-one; Determining a plurality of static pixel points and a plurality of dynamic pixel points among the plurality of second pixel points according to the plurality of first coordinates of the plurality of first pixel points and the plurality of second coordinates of the plurality of second pixel points; determining a third color value of a third static pixel point according to the second color value of the second static pixel point, where the second static pixel point is any one of the plurality of static pixel points, and the third static pixel point is a pixel point among the plurality of third pixels points corresponding to the second static pixel point; Determining a fourth color value of a third dynamic pixel point based on a first color value of a first dynamic pixel point and a second color value of a second dynamic pixel point, where the second dynamic pixel point is any one of the plurality of dynamic pixel points, the first dynamic pixel point is a pixel point in the plurality of first pixel points having the same coordinates as the second dynamic pixel point, and the third dynamic pixel point is a pixel point in the plurality of third pixel points having the same coordinates as the second dynamic pixel point; A target image is generated according to the third color value of the third static pixel and the fourth color value of the third dynamic pixel.

2. The method according to claim 1, characterized in that The second rendering information further includes a plurality of second depth values corresponding to the plurality of static pixels, and determining the third color value of the third static pixel according to the second color value of the second static pixel includes: Acquire a depth reference map, the depth reference map including a plurality of reference pixels, the plurality of reference pixels corresponding to a plurality of reference depth values in a one-to-one manner, and the plurality of reference pixels corresponding to the plurality of third pixels in a one-to-one manner; Obtaining a second depth value, a second color value, and a first motion vector of the second static pixel, where the first motion vector is related to the second motion vector, the second motion vector representing an offset position of a first coordinate of the first static pixel relative to a second coordinate of the second static pixel, the first static pixel being a pixel corresponding to the second static pixel among the plurality of first pixels; determining a third coordinate of the third static pixel point according to the second coordinate of the second static pixel point and the first motion vector; If the reference depth value corresponding to the third static pixel is equal to the second depth value of the second static pixel, the second color value of the second static pixel is used as the third color value of the third static pixel, wherein the reference depth value corresponding to the third static pixel is the reference depth value of the reference pixel having the same coordinates as the third coordinate of the third static pixel.

3. The method according to claim 2, characterized in that The obtaining of the depth reference map includes: Acquire an initial depth reference map, the initial depth reference map including a plurality of initial pixel points, the plurality of initial pixel points corresponding one-to-one to a plurality of initial depth values, and the plurality of initial pixel points corresponding one-to-one to the plurality of second pixel points; performing an update operation on the initial pixel points corresponding to the plurality of static pixel points among the plurality of initial pixel points to obtain a plurality of update values; updating the initial depth reference map according to the multiple update values to obtain the depth reference map; The update operation includes: Obtaining a second depth value and a third motion vector of a fourth static pixel, where the fourth static pixel is any one of the plurality of static pixels, the third motion vector is related to the fourth motion vector, and the fourth motion vector represents an offset position of a first coordinate of a fifth static pixel relative to the second coordinate of the fourth static pixel, where the fifth static pixel is a pixel corresponding to the fourth static pixel among the plurality of first pixels; Determining a first initial pixel point according to the second coordinate of the fourth static pixel point and the third motion vector, where the first initial pixel point is a pixel point corresponding to the fourth static pixel point among the multiple initial pixel points; If the second depth value of the fourth static pixel is less than the initial depth value corresponding to the first initial pixel, use the second depth value of the fourth static pixel as the update value corresponding to the first initial pixel; If the second depth value of the fourth static pixel is greater than or equal to the initial depth value corresponding to the first initial pixel, the initial depth value corresponding to the first initial pixel is used as the update value corresponding to the first initial pixel.

4. The method according to claim 2, characterized in that Before obtaining the second depth value, the second color value, and the first motion vector of the second static pixel, the method further includes: generating the second motion vector according to a difference between a first coordinate of the first static pixel and a second coordinate of the second static pixel; A half of the second motion vector is used as the first motion vector.

5. The method according to claim 1, wherein The determining of the fourth color value of the third dynamic pixel point according to the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point includes: Obtaining a first color value and a first identifier of the first dynamic pixel point, where the first identifier indicates whether the first dynamic pixel point is a pixel point of a person; Obtaining a second color value and a second identifier of the second dynamic pixel point, where the second identifier indicates whether the second dynamic pixel point is a pixel point of a person; If the first identifier represents that the first dynamic pixel point is a pixel point of a person, and the second identifier represents that the second dynamic pixel point is a pixel point of a person, the first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weightedly summed according to the first weight coefficient and the second weight coefficient to obtain the fourth color value of the third dynamic pixel point; the first weight coefficient corresponds to the first color value of the first dynamic pixel point, and the second weight coefficient corresponds to the second color value of the second dynamic pixel point.

6. The method according to claim 5, characterized in that The method further comprises: If the first identifier indicates that the first dynamic pixel point is not a pixel point of a person, or if the second identifier indicates that the second dynamic pixel point is not a pixel point of a person, obtaining a first depth value of the first dynamic pixel point and obtaining a second depth value of the second dynamic pixel point; The first color value of the first dynamic pixel point and the second color value of the second dynamic pixel point are weightedly summed according to the third weight coefficient and the fourth weight coefficient to obtain the fourth color value of the third dynamic pixel point; wherein, the third weight coefficient corresponds to the first color value of the first dynamic pixel point, and the fourth weight coefficient corresponds to the second color value of the second dynamic pixel point; if the first depth value of the first dynamic pixel point is greater than the second depth value of the second dynamic pixel point, the third weight coefficient is less than the fourth weight coefficient; if the first depth value of the first dynamic pixel point is less than the second depth value of the second dynamic pixel point, the third weight coefficient is greater than the fourth weight coefficient.

7. The method according to claim 1, characterized in that The determining, according to the plurality of first coordinates of the plurality of first pixel points and the plurality of second coordinates of the plurality of second pixel points, a plurality of static pixel points and a plurality of dynamic pixel points in the plurality of second pixel points comprises: determining, according to the reprojection method, a fifth motion vector of the first coordinate of the first pixel point to be classified relative to the second coordinate of the second pixel point to be classified, where the second pixel point to be classified is any one of the plurality of second pixel points, and the first pixel point to be classified is a pixel point among the plurality of first pixel points corresponding to the second pixel point to be classified; determining a sixth motion vector of the first pixel to be classified relative to the second pixel to be classified according to an optical flow method; If the difference between the fifth motion vector and the sixth motion vector is less than or equal to a preset value, determining that the second pixel to be classified is a static pixel; If the difference between the fifth motion vector and the sixth motion vector is greater than the preset value, it is determined that the second pixel to be classified is a dynamic pixel.

8. The method according to claim 7, characterized in that The first rendering information further includes a first view projection matrix, the second rendering information further includes a second view projection matrix, and the root determines, according to the reprojection method, a fifth motion vector of the first coordinate of the first pixel to be classified relative to the second coordinate of the second pixel to be classified, including: Determine the coordinates of the second pixel to be classified in the world coordinate system according to the first coordinates of the first pixel to be classified and the first view projection matrix; determining a second coordinate of the second pixel point to be classified according to the coordinates in the world coordinate system corresponding to the second pixel point to be classified and the second view projection matrix; The fifth motion vector is determined according to the first coordinates of the first pixel to be classified and the second coordinates of the second pixel to be classified.

9. The method according to claim 8, characterized in that Before determining the second coordinate of the second pixel point to be classified according to the coordinates in the world coordinate system corresponding to the second pixel point to be classified and the second view projection matrix, the method further includes: When an instruction to start drawing the main scene graph corresponding to the second rendering information is detected, the second view projection matrix is obtained.

10. The method according to claim 1, characterized in that The first rendering information is used to render a first image, the second rendering information is used to render a second image, the first moment is before the second moment, and the target image is used to be inserted between the first image and the second image.

11. An electronic device, characterized in that: The device comprises a processor configured to execute a computer program stored in a memory to implement the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images