A method, apparatus, medium, and product of image processing

Abstract

Embodiments of the present disclosure relate to a method, apparatus, medium and product of image processing. The method proposed herein includes: performing an extrapolation frame operation on a first rendered frame to generate an interpolated frame, the first rendered frame corresponding to a first virtual camera parameter of a virtual scene, the interpolated frame corresponding to a second virtual camera parameter of the virtual scene, the first rendered frame being earlier than the interpolated frame; obtaining a third virtual camera parameter corresponding to a second rendered frame before rendering the second rendered frame, the second rendered frame being later than the interpolated frame; determining a reference virtual camera parameter corresponding to the interpolated frame based on the first virtual camera parameter and the third virtual camera parameter; determining whether the interpolated frame meets an output condition based on at least a comparison of the reference virtual camera parameter and the second virtual camera parameter; and in response to the interpolated frame meeting the output condition, displaying the interpolated frame before displaying the second rendered frame. In this way, embodiments of the present disclosure can ensure the accuracy of the interpolated frame, and also ensure the logical consistency of the picture.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to image processing techniques, and more specifically, to methods, apparatus, media, and products for image processing. Background Technology

[0002] Today, image processing technology is being applied to all aspects of people's lives. It can provide diverse video and game scenes. Currently, real-time rendering applications in image processing technology have an ever-increasing demand for high-resolution images and high refresh rate displays. This poses a significant challenge to the hardware performance of traditional rasterization rendering pipelines. Summary of the Invention

[0003] In a first aspect, an image processing method is proposed. The method includes: generating an interpolated frame by performing an extrapolation operation on a first rendered frame, wherein the first rendered frame corresponds to first virtual camera parameters of a virtual scene, and the interpolated frame corresponds to second virtual camera parameters of the virtual scene, the first rendered frame being temporally earlier than the interpolated frame; acquiring third virtual camera parameters corresponding to the second rendered frame before rendering the second rendered frame, the second rendered frame being temporally later than the interpolated frame; determining reference virtual camera parameters corresponding to the interpolated frame based on the first and third virtual camera parameters; determining whether the interpolated frame satisfies an output condition based at least on a comparison of the reference and second virtual camera parameters; and displaying the interpolated frame before displaying the second rendered frame in response to the interpolated frame satisfying the output condition.

[0004] In a second aspect, an apparatus for image processing is proposed. The apparatus includes: a first execution module configured to generate an interpolated frame by performing an interpolation operation on a first rendered frame, the first rendered frame corresponding to first virtual camera parameters of a virtual scene, and the interpolated frame corresponding to second virtual camera parameters of the virtual scene, wherein the first rendered frame is temporally earlier than the interpolated frame; an acquisition module configured to acquire third virtual camera parameters corresponding to a second rendered frame before rendering the second rendered frame, wherein the second rendered frame is temporally later than the interpolated frame; a first determination module configured to determine reference virtual camera parameters corresponding to the interpolated frame based on the first virtual camera parameters and the third virtual camera parameters; a second determination module configured to determine whether the interpolated frame satisfies an output condition based at least on a comparison of the reference virtual camera parameters and the second virtual camera parameters; and a display module configured to display the interpolated frame before displaying the second rendered frame in response to the interpolated frame satisfying the output condition.

[0005] In a third aspect, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] In this way, the embodiments of this disclosure can compare the reference virtual camera parameters of the second rendering frame with the second virtual camera parameters of the interpolated frame, which can ensure the accuracy of the interpolated frame, avoid object drift, viewpoint misalignment and other situations, and ensure the logical consistency of the picture.

[0009] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0011] Figure 1 A schematic diagram of an example environment according to an embodiment of the present disclosure is shown.

[0012] Figure 2 An example flowchart of an image processing method according to some embodiments of the present disclosure is shown.

[0013] Figure 3 An example structural block diagram of an example apparatus for image processing according to some embodiments of the present disclosure is shown.

[0014] Figure 4 A schematic structural block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0015] Throughout all the accompanying figures, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0016] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0017] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0018] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0019] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0021] As mentioned above, image processing technology is now being applied to all aspects of people's lives. It can provide a variety of video and game scenes. Currently, real-time rendering applications in image processing technology have an ever-increasing demand for high-resolution images and high refresh rate displays. This poses a serious challenge to the hardware performance of traditional rasterization rendering pipelines.

[0022] Embodiments of this disclosure propose an image processing scheme. The scheme includes: generating an interpolated frame by performing an interpolation operation on a first rendered frame, wherein the first rendered frame corresponds to first virtual camera parameters of a virtual scene, and the interpolated frame corresponds to second virtual camera parameters of the virtual scene, the first rendered frame being earlier in time than the interpolated frame; obtaining third virtual camera parameters corresponding to the second rendered frame before rendering a second rendered frame, the second rendered frame being later in time than the interpolated frame; determining reference virtual camera parameters corresponding to the interpolated frame based on the first virtual camera parameters and the third virtual camera parameters; determining whether the interpolated frame satisfies an output condition based at least on a comparison between the reference virtual camera parameters and the second virtual camera parameters; and displaying the interpolated frame before displaying the second rendered frame in response to the interpolated frame satisfying the output condition.

[0023] In this way, embodiments of this disclosure generate interpolated frames through extrapolation operations, outputting the interpolated frames without waiting for the second rendering frame to complete, thereby significantly reducing screen latency. Furthermore, embodiments of this disclosure can compare the reference virtual camera parameters of the second rendering frame with the second virtual camera parameters of the interpolated frame, ensuring the accuracy of the interpolated frames and avoiding issues such as object drift and perspective misalignment, thus guaranteeing logical consistency in the visuals. Additionally, embodiments of this disclosure only display the interpolated frames after the conditions are met, ensuring the accuracy of the output rendering frames, thereby improving visual smoothness and reliability.

[0024] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0025] Example environment:

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110.

[0027] In this example environment 100, electronic device 110 can run an application 120 that supports image processing. Application 120 can be any suitable type of application for image processing, examples of which may include, but are not limited to, game applications, video applications, scene simulation applications, or other suitable applications. User 140 can interact with application 120 via electronic device 110 and / or its attached devices.

[0028] exist Figure 1 In environment 100, if application 120 is active, electronic device 110 can use application 120 to present interface 150 for supporting image processing. Such interface 150 can, for example, present a virtual scene in a game application.

[0029] In some embodiments, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0030] In some embodiments, the image processing method disclosed herein can be executed by electronic device 110, server 130, or jointly by electronic device 110 and server 130. If the image processing method is executed by server 130, then after generating the interpolated frame, server 130 can encode the interpolated frame to generate a bitstream. Server 130 can transmit the bitstream to electronic device 110 so that electronic device 110 can decode the bitstream and present corresponding content (e.g., a virtual scene) on interface 150.

[0031] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for image processing applications 120 in electronic devices 110.

[0032] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, server 130 and electronic device 110 can achieve signaling interaction through the communication connection between them.

[0033] Some exemplary embodiments of this disclosure will be described in detail below. It should be noted that section headings are used in this document for ease of understanding and not to limit the embodiments disclosed in a section to that section. Furthermore, although some embodiments are described with reference to example scenario 100, the disclosed techniques are also applicable to other image processing techniques.

[0034] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0035] The following description will continue with reference to the accompanying drawings to illustrate some exemplary embodiments of this disclosure.

[0036] Example process:

[0037] Figure 2 A flowchart of an example process 200 for image processing according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 110, at server 130, or jointly executed by electronic device 110 and server 130. Reference will be made below. Figure 1 The example environment 100 shown illustrates process 200 as being implemented at electronic device 110.

[0038] As mentioned above, real-time rendering applications in image processing technology increasingly demand high-resolution images and high refresh rate displays. This poses a severe challenge to the hardware performance of traditional rasterization rendering pipelines. Traditionally, interpolated frame generation techniques are used to improve visual smoothness. However, traditional interpolated frame generation techniques introduce various problems. For example, traditional interpolated frame generation techniques can lead to an inherent contradiction between end-to-end latency and visual smoothness. Furthermore, in highly dynamic, low-latency interactive application scenarios, traditional interpolated frame generation techniques still have significant deficiencies in response speed, logical consistency, and failure handling mechanisms, failing to meet the dual requirements of low-latency applications and high-fidelity simulations for both low latency and high reliability.

[0039] To address the aforementioned problems, this disclosure provides an image processing method, with reference to... Figure 2In step 210, the electronic device 110 generates an interpolated frame by performing an extrapolation operation on the first rendered frame. The first rendered frame corresponds to the first virtual camera parameters of the virtual scene. The interpolated frame corresponds to the second virtual camera parameters of the virtual scene. The first rendered frame is earlier than the interpolated frame in time.

[0040] As an example, before generating the interpolated frame, the electronic device 110 can acquire the first rendered frame. The output time of the first rendered frame can be, for example, a first moment (e.g., the 10th second). The electronic device 110 can generate an interpolated frame between the first and second rendered frames based on the first rendered frame before the second rendered frame is generated. The output time of the second rendered frame can, for example, correspond to a second moment (e.g., the 11th second). The output time of the interpolated frame can, for example, correspond to a third moment (e.g., the 10.5th second).

[0041] Taking a game scene as an example, when generating interpolated frames, electronic device 110 can, for instance, acquire the first rendered frame at the 10th second. Electronic device 110 can analyze the first rendered frame to determine the first virtual camera parameters and motion vectors used to generate the interpolated frames. These first virtual camera parameters may include, for example, the virtual camera's position coordinates in the world coordinate system, field of view, and rotation quaternions. After obtaining the first virtual camera parameters and motion vectors, electronic device 110 can perform extrapolation operations based on these parameters to generate the interpolated frames.

[0042] In some embodiments, the electronic device 110 may, in response to completing the rendering of a first rendered frame, determine second virtual camera parameters corresponding to an interpolated frame based on first virtual camera parameters of the first rendered frame and historical virtual camera parameters of at least one historical frame. As an example, the electronic device 110 may determine the second virtual camera parameters not only based on the first virtual camera parameters, but also based on both the first and historical virtual camera parameters to ensure the accuracy of the second virtual camera parameters. The electronic device 110 may utilize an appropriate algorithm to determine the second virtual camera parameters corresponding to the virtual scene at 10.5 seconds. For example, the electronic device 110 may utilize a high-order motion model, assuming the virtual camera maintains an inertial motion trend, and fit a camera trajectory based on the first virtual camera parameters to determine the second virtual camera parameters. The electronic device 110 may also utilize an appropriate prediction model to predict the second virtual camera parameters based on the first virtual camera parameters.

[0043] After determining the parameters of the second virtual camera, the electronic device 110 can perform projection operations on different objects in the virtual scene to generate interpolated frames. Specifically, the electronic device 110 can project a first pixel corresponding to a static object onto a first position in the interpolated frame based on the second virtual camera parameters. Further, the electronic device 110 can project a second pixel of a dynamic object onto a second position in the interpolated frame based on the motion vector of the second pixel and the second virtual camera parameters.

[0044] As an example, the second virtual camera parameters can be used to determine the position of the first pixel (e.g., a static pixel) in the interpolated frame. Motion vectors can be used to determine the position of the second pixel (e.g., a dynamic pixel) in the interpolated frame. Static pixels can correspond to static objects (e.g., background, buildings, etc.) in the virtual scene. Dynamic pixels can correspond to dynamic objects (e.g., people, vehicles, etc.) in the virtual scene.

[0045] When performing interpolation, the electronic device 110 can perform pixel reprojection operations based on the aforementioned information (including second virtual camera parameters and motion vectors). For static objects in the virtual scene, the electronic device 110 can map static pixels to the first position of the interpolated frame based on the second virtual camera parameters using the camera's inverse view transformation matrix. For dynamic objects in the virtual scene, the electronic device 110 can determine the displacement of moving pixels based on motion vector information and map the moving pixels to the second position of the interpolated frame based on camera motion (e.g., viewpoint transformation, position transformation, etc.). After the pixel reprojection operation is completed, the electronic device 110 can obtain the interpolated frame.

[0046] In this way, embodiments of the present disclosure can perform different processing for different objects in a virtual scene, thereby improving the generation quality of interpolated frames.

[0047] After generating the interpolated frame, there may be blank, holed regions within the frame. The electronic device 110 can fill these holed regions to ensure the integrity of the interpolated frame. Specifically, in response to the interpolated frame including a first holed region, the electronic device 110 can determine at least one pixel associated with the holed region from at least one historical frame. The electronic device 110 can determine the pixel value of the first holed region based on at least one pixel.

[0048] As an example, electronic device 110 can acquire at least one historical frame. This at least one historical frame may, for example, include a third rendered frame. The third rendered frame is temporally earlier than the first rendered frame. For example, the output time of the first rendered frame may be 10 seconds, and the output time of the third rendered frame may be 8 seconds. Due to parallax effects, the third rendered frame, which precedes the first rendered frame, typically includes background information occluded by the foreground in the first rendered frame. Therefore, electronic device 110 can perform hole filling on the interpolated frame based on the third rendered frame. Specifically, electronic device 110 can determine at least one pixel in the third rendered frame associated with the first hole region. Further, electronic device 110 can perform a spatiotemporal transformation mapping based on at least one pixel, mapping the at least one pixel to the first hole region of the interpolated frame for hole filling.

[0049] Taking a racing game scene as an example, the hole-filling process is described. After rendering frame 100 (the first rendering frame), electronic device 110 can generate frame 101 (the interpolated frame). The road surface obscured by the race car in frame 100 will be revealed in frame 101 as the race car moves forward, thus forming a blank hole area behind the rear of the car (the first hole area). For this hole area, electronic device 110 can obtain the rendered frame 98 (at least one historical frame, also the third rendering frame). Due to the parallax effect, the race car is positioned further back in frame 98, and the target road surface texture that is not obscured by the race car at this time can correspond to the hole area. Electronic device 110 can map the pixels corresponding to the target road surface texture to the hole area in frame 101 to fill the hole in frame 101.

[0050] In this way, the embodiments of this disclosure can repair holes in the interpolated frame based on historical frames, thereby effectively ensuring the integrity of the image in the interpolated frame and providing users with better visual effects after outputting the interpolated frame.

[0051] In some embodiments, the electronic device 110 may, in response to the interpolation frame including a second hole region, utilize a patching model to generate fill pixels for the second hole region based on multiple existing pixels in the interpolation frame. As an example, after filling the first hole region, a second hole region may still exist in the interpolation frame. In this case, the electronic device 110 may, for example, utilize an image content-based inpainting algorithm to patch the second hole region based on multiple existing pixels in the interpolation frame to generate the interpolation frame.

[0052] In this way, the embodiments of this disclosure can perform secondary repair on the interpolated frame, thereby further ensuring the integrity of the image in the interpolated frame.

[0053] In step 220, before rendering the second rendering frame, the electronic device 110 obtains the third virtual camera parameters corresponding to the second rendering frame. The second rendering frame is temporally later than the interpolation frame. As an example, in the rendering pipeline of a game engine, the processing progress of the Central Processing Unit (CPU) is ahead of that of the Graphics Processing Unit (GPU). Before rendering the second rendering frame, the CPU has completed the physical simulation and scene culling of the second rendering frame, and determined the third virtual camera parameters of the virtual camera at the second moment. Therefore, the electronic device 110 can read the third virtual camera parameters corresponding to the second rendering frame determined by the CPU before rendering the second rendering frame.

[0054] In step 230, the electronic device 110 determines the reference virtual camera parameters corresponding to the interpolated frame based on the first virtual camera parameters and the third virtual camera parameters. As an example, after the third virtual camera parameters are acquired, the electronic device 110 can determine the reference virtual camera parameters at the third time step based on the first and third virtual camera parameters. These reference virtual camera parameters can, for example, indicate the actual state of the virtual camera at the third time step (e.g., actual position, actual viewpoint, etc.).

[0055] In step 240, the electronic device 110 determines whether the interpolated frame meets the output conditions, based at least on a comparison of the reference virtual camera parameters and the second virtual camera parameters. As an example, after determining the reference virtual camera parameters, the electronic device 110 can compare the reference virtual camera parameters and the second virtual camera parameters using an appropriate algorithm to determine the differences between them. These differences include at least one of the following: the distance between the first position indicated by the reference virtual camera parameters and the second position indicated by the second virtual camera parameters; and the angular difference between the first orientation indicated by the reference virtual camera parameters and the second orientation indicated by the second virtual camera parameters.

[0056] For example, the electronic device 110 can perform a geometric consistency comparison of the reference virtual camera parameters and the second virtual camera parameters, calculate the Euclidean distance and viewing angle deviation of the virtual camera position in the interpolated frame, and determine whether the interpolated frame meets the output conditions based on the Euclidean distance and viewing angle deviation.

[0057] When determining whether an interpolated frame meets the output conditions, the electronic device 110 can compare not only the parameters of the reference virtual camera and the parameters of the second virtual camera, but also detect whether the motion trajectory of dynamic motion in the virtual scene undergoes abrupt changes to determine the accuracy of the interpolated frame. First, the electronic device 110 can identify at least one object from multiple dynamic objects in the interpolated frame as a judgment benchmark. Specifically, the electronic device 110 can detect multiple dynamic objects from the interpolated frame. Further, the electronic device 110 can identify at least one object from the multiple dynamic objects.

[0058] As an example, the electronic device 110 can parse the interpolated frame to determine multiple dynamic objects in the interpolated frame. These multiple dynamic objects may include, for example, a character, an animal, a moving object, a vehicle controlled by a character, etc. Furthermore, the electronic device 110 can determine at least one object from the multiple dynamic objects. This at least one object may be referred to as a key object. A key object may, for example, be a character or a vehicle controlled by a character.

[0059] In some embodiments, after identifying at least one object, the electronic device 110 can determine whether an interpolated frame satisfies an output condition based on the motion trajectory of the at least one object. Specifically, the electronic device 110 can determine the motion trajectory of at least one object based on the interpolated frame in response to the difference between the reference virtual camera parameters and the second virtual camera parameters being less than or equal to a threshold. Further, the electronic device 110 can determine that the interpolated frame satisfies the output condition in response to the motion trajectory satisfying motion constraints.

[0060] As an example, after the difference between the reference virtual camera parameters and the second virtual camera parameters is less than or equal to a threshold, the electronic device 110 can utilize the central processing unit's logic processing, which precedes the graphics processing unit's features, to obtain reference data for at least one object (e.g., a character) in the second rendering frame. This reference data may include, for example, information such as the character's reference coordinates and reference orientation in the second rendering frame. Based on the reference data and the character's historical data (e.g., historical coordinates and historical orientation) in historical rendering frames, the electronic device 110 can determine the character's reference motion trajectory.

[0061] Furthermore, the electronic device 110 can determine the bounding box of the character in the interpolated frame. Based on the bounding box, the electronic device 110 can determine the target data of the character in the interpolated frame, which may include information such as the target coordinates and target orientation of the character in the interpolated frame. Based on historical data and target data in historical rendering frames, the electronic device 110 can determine the target motion trajectory of the character. Based on the reference motion trajectory, the electronic device 110 determines whether the target motion trajectory satisfies motion constraints. For example, it determines whether the character in the interpolated frame conforms to the continuity assumption. For example, whether the target coordinates or target orientation of the character in the interpolated frame can be connected with the reference coordinates or reference orientation of the character in the second rendering frame. If it is determined that the character conforms to the continuity assumption, then the electronic device 110 can determine that the interpolated frame meets the output conditions.

[0062] Taking a racing game scene as an example, the car appears to have moved forward 10 meters in the second rendered frame. However, in the interpolated frame, the car appears to have moved forward 5 meters. Since 5 meters falls between 10 meters and 0 meters, and the car's orientation remains consistent, the electronic device 110 can determine that the interpolated frame meets the output conditions.

[0063] In this way, the embodiments of this disclosure can determine the accuracy of the interpolated frame through multiple checks. After determining that the interpolated frame is accurate, the interpolated frame is then output, which can effectively improve the continuity of video or game scenes.

[0064] In step 250, in response to the interpolation frame satisfying the output conditions, the electronic device 110 displays the interpolation frame before displaying the second rendered frame. As an example, the electronic device 110 may provide the interpolation frame to the output device when it is determined that the difference between the reference virtual camera parameters and the second virtual camera parameters is less than a threshold, and the motion trajectory of at least one object in the interpolation frame satisfies motion constraints. Upon receiving the interpolation frame, the output device may output the interpolation frame before outputting the second rendered frame. The output device may, for example, include a suitable display, and may be connected to the electronic device 110.

[0065] In some embodiments, the electronic device 110 may determine that the interpolated frame does not meet the output conditions in response to the difference between the reference virtual camera parameters and the second virtual camera parameters reaching a threshold. As an example, the electronic device 110 may determine whether the interpolated frame meets the output conditions based on the aforementioned Euclidean distance and viewing angle deviation. For instance, if the Euclidean distance reaches a first threshold and the viewing angle deviation reaches a second threshold, then the interpolated frame does not meet the output conditions.

[0066] Taking a racing game scene as an example, the player suddenly swerves to the left and brakes at the very moment the 100th frame finishes rendering. However, the generated 101st frame (e.g., an interpolated frame) shows the car moving forward in the direction shown in the 100th frame. However, after acquiring the data for the 102nd frame, the electronic device 110 can determine that the actual camera angle in the 102nd frame has shifted 30 degrees to the left, and the position has changed drastically. The electronic device 110 determines that there is a significant discrepancy between the actual camera parameters and the predicted camera parameters of the interpolated frame. If the interpolated frame continues to be output at this point, the player will see the car move forward first and then suddenly jump to the left, creating a severe sense of distortion. Therefore, the electronic device 110 can determine that the interpolated frame does not meet the output conditions.

[0067] In some embodiments, the electronic device 110 may discard an interpolated frame in response to the interpolated frame not meeting the output conditions. Further, the electronic device 110 applies dynamic blur processing to a second rendered frame based on a first rendered frame. The electronic device 110 may also display the processed second rendered frame.

[0068] As an example, when an interpolated frame does not meet the output conditions, the electronic device 110 can discard the output interpolated frame and delete it from the video memory to avoid severe visual artifacts or screen tearing. To compensate for the loss of visual smoothness caused by discarding the output interpolated frame, the electronic device 110 can send instructions to the graphics rendering pipeline that is preparing to render the second rendering frame to dynamically adjust its motion blur rendering parameters. For example, the electronic device 110 can calculate the duration from the end of the first rendering frame to the end of the second rendering frame. Based on this duration, the electronic device 110 can adjust the shutter integration time or blur vector intensity of the second rendering frame, thereby enabling a more continuous transition between the first and second rendering frames. Furthermore, the electronic device 110 can output the processed second rendering frame through a display device.

[0069] In this way, embodiments of this disclosure generate interpolated frames through extrapolation operations, outputting the interpolated frames without waiting for the second rendering frame to complete, thereby significantly reducing screen latency. Furthermore, embodiments of this disclosure can compare the reference virtual camera parameters of the second rendering frame with the second virtual camera parameters of the interpolated frame, ensuring the accuracy of the interpolated frames and avoiding issues such as object drift and perspective misalignment, thus guaranteeing logical consistency in the visuals. Additionally, embodiments of this disclosure only display the interpolated frames after the conditions are met, ensuring the accuracy of the output rendering frames, thereby improving visual smoothness and reliability.

[0070] Example devices and equipment:

[0071] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 3 A schematic structural diagram of an example device 300 for graphics processing is shown, according to some scenarios. Device 300 may be implemented as or included in electronic device 110. The various modules / components in device 300 may be implemented by hardware, software, firmware, or any combination thereof.

[0072] like Figure 3 As shown, the apparatus 300 includes: a first execution module 310 configured to generate an interpolated frame by performing an interpolation operation on a first rendered frame, wherein the first rendered frame corresponds to a first virtual camera parameter of a virtual scene, and the interpolated frame corresponds to a second virtual camera parameter of the virtual scene, wherein the first rendered frame is earlier than the interpolated frame in time; an acquisition module 320 configured to acquire a third virtual camera parameter corresponding to the second rendered frame before rendering the second rendered frame, wherein the second rendered frame is later than the interpolated frame in time; a first determination module 330 configured to determine a reference virtual camera parameter corresponding to the interpolated frame based on the first virtual camera parameter and the third virtual camera parameter; a second determination module 340 configured to determine whether the interpolated frame satisfies an output condition based at least on a comparison between the reference virtual camera parameter and the second virtual camera parameter; and a display module 350 configured to display the interpolated frame before displaying the second rendered frame in response to the interpolated frame satisfying the output condition.

[0073] In some embodiments, the first execution module 310 is further configured to: in response to completing the rendering of the first rendering frame, determine the second virtual camera parameters corresponding to the interpolation frame based on the first virtual camera parameters of the first rendering frame and the historical virtual camera parameters of at least one historical frame; for a first pixel corresponding to a static object, project the first pixel to a first position in the interpolation frame based on the second virtual camera parameters; and for a second pixel corresponding to a dynamic object, project the second pixel to a second position in the interpolation frame based on the motion vector of the second pixel and the second virtual camera parameters.

[0074] In some embodiments, the apparatus 300 further includes a first filling module configured to, in response to the interpolation frame including a first hole region, determine at least one pixel associated with the first hole region from the at least one historical frame; and determine a pixel value of the first hole region based on the at least one pixel.

[0075] In some embodiments, the apparatus 300 further includes a second filling module configured to, in response to the interpolation frame including a second hole region, generate filling pixels for the second hole region based on a plurality of existing pixels in the interpolation frame using a patching model.

[0076] In some embodiments, the second determining module 340 is further configured to: determine that the interpolated frame does not meet the output condition in response to the difference between the reference virtual camera parameters and the second virtual camera parameters reaching a threshold.

[0077] In some embodiments, the difference includes at least one of the following: the distance between the first position indicated by the reference virtual camera parameters and the second position indicated by the second virtual camera parameters; the angular difference between the first orientation indicated by the reference virtual camera parameters and the second orientation indicated by the second virtual camera parameters.

[0078] In some embodiments, the second determining module 340 is further configured to: determine the motion trajectory of at least one object based on the interpolated frame in response to the difference between the reference virtual camera parameters and the second virtual camera parameters being less than or equal to the threshold; and determine that the interpolated frame satisfies the output condition in response to the motion trajectory satisfying the motion constraint.

[0079] In some embodiments, the apparatus 300 further includes a detection module configured to detect a plurality of dynamic objects from the interpolated frame; and to determine the at least one object from the plurality of dynamic objects.

[0080] In some embodiments, the apparatus 300 further includes a processing module configured to discard the interpolated frame in response to the interpolated frame not meeting the output condition; apply dynamic blur processing to the second rendered frame based on the first rendered frame; and display the processed second rendered frame.

[0081] Figure 4 A block diagram of a computing device 400 in which various embodiments of the present disclosure may be implemented is shown. The computing device 400 may be implemented as an electronic device 110, or may be included in an electronic device 110.

[0082] It should be understood that, Figure 4 The computing device 400 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0083] like Figure 4 As shown, computing device 400 includes general-purpose computing device 400. Computing device 400 may include at least one or more processors or processing units 410, memory 420, storage unit 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460.

[0084] In some embodiments, the computing device 400 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, a large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 400 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0085] Processing unit 410 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 420. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 400. Processing unit 410 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0086] Computing device 400 typically includes various computer storage media. Such media can be any media accessible by computing device 400, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 420 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 430 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 400.

[0087] The computing device 400 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 4 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0088] Communication unit 440 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in computing device 400 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, computing device 400 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0089] Input device 450 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 460 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 440, computing device 400 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 400 can also communicate with one or more devices that enable a user to interact with computing device 400, or, if necessary, with any device that enables computing device 400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via an input / output (I / O) interface (not shown).

[0090] In some embodiments, some or all of the components of computing device 400 may be arranged in a cloud computing architecture, rather than integrated into a single device. In a cloud computing architecture, components may be remotely provided and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (WAN), such as the Internet, using suitable protocols. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across remote data center locations. Cloud computing infrastructure may provide services through shared data centers, although they appear as a single access point to users. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from service providers at remote locations. Alternatively, the components and functionality described herein may be provided by conventional servers or installed directly or otherwise on client devices.

[0091] In embodiments of this disclosure, computing device 400 may be used to implement image processing. Memory 420 may include one or more image processing modules 425 having one or more program instructions. These modules are accessible and executable by processing unit 410 to perform the functions of the various embodiments described herein.

[0092] In an example embodiment of image processing, the virtual scene may be processed, for example, by the image processing module 425 to generate a picture of the virtual scene or a bitstream corresponding to the picture of the virtual scene. The picture or bitstream of the virtual scene may be provided as output 470 via output device 460.

[0093] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. An image processing method, characterized in that, The method includes: An interpolated frame is generated by performing an extrapolation operation on the first rendered frame. The first rendered frame corresponds to the first virtual camera parameters of the virtual scene, and the interpolated frame corresponds to the second virtual camera parameters of the virtual scene. The first rendered frame is earlier than the interpolated frame in time. Before rendering the second rendering frame, the parameters of the third virtual camera corresponding to the second rendering frame are obtained, wherein the second rendering frame is later in time than the interpolation frame. Based on the first virtual camera parameters and the third virtual camera parameters, determine the reference virtual camera parameters corresponding to the interpolated frame; Based at least on a comparison of the reference virtual camera parameters and the second virtual camera parameters, it is determined whether the interpolated frame satisfies the output conditions; and In response to the interpolation frame satisfying the output condition, the interpolation frame is displayed before the second rendering frame is displayed; The process of generating an interpolated frame by performing an extrapolation frame operation on the first rendered frame includes: In response to the completion of rendering the first rendered frame, the second virtual camera parameters corresponding to the interpolated frame are determined based on the first virtual camera parameters of the first rendered frame and the historical virtual camera parameters of at least one historical frame. For the first pixel corresponding to the static object, based on the second virtual camera parameters, the first pixel is projected onto the first position in the interpolated frame; and For the second pixel corresponding to the dynamic object, based on the motion vector of the second pixel and the second virtual camera parameters, the second pixel is projected to the second position in the interpolated frame.

2. The method according to claim 1, characterized in that, The method further includes: In response to the interpolated frame including a first hole region, at least one pixel associated with the first hole region is determined from the at least one historical frame; and Based on the at least one pixel, determine the pixel value of the first hole region.

3. The method according to claim 1, characterized in that, The method further includes: In response to the interpolated frame including a second hole region, a patching model is used to generate fill pixels for the second hole region based on multiple existing pixels in the interpolated frame.

4. The method according to claim 1, characterized in that, Determining whether the interpolated frame satisfies the output conditions, based at least on a comparison of the reference virtual camera parameters and the second virtual camera parameters, includes: In response to the difference between the reference virtual camera parameters and the second virtual camera parameters reaching a threshold, it is determined that the interpolated frame does not meet the output condition.

5. The method according to claim 4, characterized in that, The difference includes at least one of the following: The distance between the first position indicated by the reference virtual camera parameters and the second position indicated by the second virtual camera parameters; The angular difference between the first orientation indicated by the reference virtual camera parameters and the second orientation indicated by the second virtual camera parameters.

6. The method according to claim 4, characterized in that, Determining whether the interpolated frame satisfies the output conditions, based at least on a comparison of the reference virtual camera parameters and the second virtual camera parameters, includes: In response to the difference between the reference virtual camera parameters and the second virtual camera parameters being less than the threshold, the motion trajectory of at least one object is determined based on the interpolated frame; and In response to the motion trajectory satisfying the motion constraints, it is determined that the interpolated frame satisfies the output conditions.

7. The method according to claim 6, characterized in that, The method further includes: Detect multiple dynamic objects from the interpolated frame; and From the plurality of dynamic objects, determine at least one object.

8. The method according to claim 1, characterized in that, The method further includes: In response to the interpolation frame not meeting the output condition, the interpolation frame is discarded; Based on the first rendered frame, apply dynamic blur processing to the second rendered frame; and Displays the processed second rendered frame.

9. An apparatus for image processing, characterized in that, The device includes: The first execution module is configured to generate an interpolated frame by performing an interpolation frame operation on the first rendered frame. The first rendered frame corresponds to the first virtual camera parameters of the virtual scene, and the interpolated frame corresponds to the second virtual camera parameters of the virtual scene. The first rendered frame is earlier than the interpolated frame in time. The acquisition module is configured to acquire the third virtual camera parameters corresponding to the second rendering frame before rendering the second rendering frame, wherein the second rendering frame is later in time than the interpolation frame. The first determining module is configured to determine reference virtual camera parameters corresponding to the interpolated frame based on the first virtual camera parameters and the third virtual camera parameters. The second determining module is configured to determine whether the interpolated frame satisfies the output conditions based at least on a comparison of the reference virtual camera parameters and the second virtual camera parameters; and The display module is configured to display the interpolation frame before displaying the second rendering frame in response to the interpolation frame satisfying the output condition. The first execution module is further configured to: in response to completing the rendering of the first rendering frame, determine the second virtual camera parameters corresponding to the interpolation frame based on the first virtual camera parameters of the first rendering frame and the historical virtual camera parameters of at least one historical frame; for a first pixel corresponding to a static object, project the first pixel to a first position in the interpolation frame based on the second virtual camera parameters; and for a second pixel corresponding to a dynamic object, project the second pixel to a second position in the interpolation frame based on the motion vector of the second pixel and the second virtual camera parameters.

10. An electronic device, characterized in that, The device includes: At least one processor; and At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processor.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.

Images