A method, apparatus, medium, and product of image processing

Abstract

Embodiments of the present disclosure relate to a method, apparatus, medium and product of image processing. The method proposed herein comprises: in response to a rendering system being in a stable state, performing rendering of a first rendering cycle with a first rendering plan, the first rendering plan indicating displaying at least one first interpolation frame between a first start rendering frame and a first end rendering frame; in response to the rendering of the second end rendering frame not being completed at a preset time of a second rendering cycle, performing rendering of the second rendering cycle with a second rendering plan, the second rendering plan indicating displaying at least one second interpolation frame between a second start rendering frame and a second end rendering frame; and after the second rendering cycle ends, performing rendering of a third rendering cycle with a third rendering plan, the third rendering plan indicating increasing the number of interpolation frames between a third start rendering frame and a third end rendering frame. In this way, embodiments of the present disclosure can guarantee visual coherence.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to image processing techniques, and more specifically, to methods, apparatus, media, and products for image processing. Background Technology

[0002] Today, image processing technology is being applied to all aspects of people's lives. It provides diverse video and game scenes. Currently, high frame rates and high image quality remain core user demands. Vertical synchronization (VSync) technology is widely used to eliminate screen tearing. However, the rendering load of modern game scenes is highly dynamic, involving complex geometric calculations or texture processing, which often leads to fluctuations in single-frame rendering time, resulting in a decrease in image quality. Summary of the Invention

[0003] In a first aspect, an image processing method is proposed. The method includes: in response to a rendering system being in a stable state, executing rendering of a first rendering cycle according to a first rendering plan, the first rendering plan indicating that at least one first interpolated frame is displayed between a first start rendering frame and a first end rendering frame, the at least one first interpolated frame including at least an interpolated frame generated based on the first end rendering frame; in response to a second rendering cycle not being completed at a preset time of the second rendering cycle, executing rendering of a second rendering cycle according to a second rendering plan, the second rendering plan indicating that at least one second interpolated frame is displayed between the second start rendering frame and the second end rendering frame, the at least one second interpolated frame being an extrapolated frame; and after the second rendering cycle ends, executing rendering of a third rendering cycle according to a third rendering plan, the third rendering plan indicating that the number of interpolated frames between a third start rendering frame and a third end rendering frame is increased such that the duration of the third rendering cycle is longer than that of the first and second rendering cycles.

[0004] In a second aspect, an apparatus for image processing is proposed. The apparatus includes: a first execution module configured to execute rendering of a first rendering cycle according to a first rendering plan in response to a rendering system being in a stable state, the first rendering plan indicating that at least one first interpolated frame is displayed between a first start rendering frame and a first end rendering frame, the at least one first interpolated frame including at least an interpolated frame generated based on the first end rendering frame; a second execution module configured to execute rendering of a second rendering cycle according to a second rendering plan in response to the rendering of a second end rendering frame not being completed at a preset time in the second rendering cycle, the second rendering plan indicating that at least one second interpolated frame is displayed between the second start rendering frame and the second end rendering frame, the at least one second interpolated frame being an extrapolated frame; and a third execution module configured to execute rendering of a third rendering cycle according to a third rendering plan after the second rendering cycle has ended, the third rendering plan indicating that the number of interpolated frames between a third start rendering frame and a third end rendering frame is increased such that the duration of the third rendering cycle is longer than that of the first and second rendering cycles.

[0005] In a third aspect, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] In this way, the embodiments of this disclosure can ensure visual continuity and enable the rendering pipeline to transition more smoothly from an abnormal state to a stable state.

[0009] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0011] Figure 1 A schematic diagram of an example environment according to some embodiments of the present disclosure is shown;

[0012] Figure 2 An example flowchart of an image processing method according to some embodiments of the present disclosure is shown;

[0013] Figure 3 Example timing diagrams of image processing according to some embodiments of the present disclosure are shown;

[0014] Figure 4 A schematic structural block diagram of an example apparatus for image processing according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0016] Throughout all the accompanying figures, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0017] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0018] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0019] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0020] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0022] As mentioned above, image processing technology is now applied to all aspects of people's lives. It provides diverse video and game scenes. Currently, high frame rates and high image quality remain core user demands. Vertical synchronization (VSync) technology is widely used to eliminate screen tearing. However, the rendering load of modern game scenes is highly dynamic, involving complex geometric calculations or texture processing, which often leads to fluctuations in single-frame rendering time, resulting in a decrease in image quality.

[0023] Embodiments of this disclosure propose an image processing scheme. The scheme includes: in response to a rendering system being in a stable state, executing rendering for a first rendering cycle according to a first rendering plan, the first rendering plan indicating the display of at least one first interpolated frame between a first start rendering frame and a first end rendering frame, the at least one first interpolated frame including at least an interpolated frame generated based on the first end rendering frame; in response to a second rendering cycle not being completed at a preset time, executing rendering for the second rendering cycle according to a second rendering plan, the second rendering plan indicating the display of at least one second interpolated frame between a second start rendering frame and a second end rendering frame, the at least one second interpolated frame being an extrapolated frame; and after the second rendering cycle ends, executing rendering for a third rendering cycle according to a third rendering plan, the third rendering plan indicating an increase in the number of interpolated frames between a third start rendering frame and a third end rendering frame, such that the duration of the third rendering cycle is longer than that of the first rendering cycle and the second rendering cycle.

[0024] In this way, the embodiments of this disclosure enhance the robustness of the real-time rendering system in a vertically synchronized environment, eliminating screen stuttering caused by rendering time fluctuations. By switching between interpolation and extrapolation modes, an interpolated frame can be generated based on the current frame, filling in the gaps caused by rendering latency. This avoids screen freezing or repeated display due to dropped frames, ensuring visual continuity. Furthermore, the embodiments of this disclosure can restore the stability of the rendering pipeline by executing a third rendering plan. This allows the display rhythm of the vertically synchronized environment to be met without discarding any critical rendering tasks, and the frame queue required for interpolation to be refilled, achieving a smoother transition of the rendering pipeline from an abnormal state to a stable state.

[0025] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0026] Example Environment

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110.

[0028] In this example environment 100, electronic device 110 can run an application 120 that supports image processing. Application 120 can be any suitable type of application for image processing, examples of which may include, but are not limited to, game applications, video applications, scene simulation applications, or other suitable applications. User 140 can interact with application 120 via electronic device 110 and / or its attached devices.

[0029] exist Figure 1 In environment 100, if application 120 is active, electronic device 110 can use application 120 to present interface 150 for supporting image processing. Such interface 150 can, for example, present a virtual scene in a game application.

[0030] In some embodiments, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0031] In some embodiments, the image processing method disclosed herein can be executed by electronic device 110, server 130, or jointly by electronic device 110 and server 130. If the image processing method is executed by server 130, after generating rendering frames and / or interpolation frames, server 130 can encode the rendering frames and / or interpolation frames to generate a bitstream. Server 130 can transmit the bitstream to electronic device 110 so that electronic device 110 can decode the bitstream and present corresponding content (e.g., a virtual scene) on interface 150.

[0032] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for image processing applications 120 in electronic devices 110.

[0033] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, server 130 and electronic device 110 can achieve signaling interaction through the communication connection between them.

[0034] Some exemplary embodiments of this disclosure will be described in detail below. It should be noted that section headings are used in this document for ease of understanding and not to limit the embodiments disclosed in a section to that section. Furthermore, although some embodiments are described with reference to example scenario 100, the disclosed techniques are also applicable to other image processing techniques.

[0035] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0036] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0037] Example process

[0038] Figure 2 A flowchart of an example process 200 for image processing according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 110, at server 130, or jointly executed by electronic device 110 and server 130. Reference will be made below. Figure 1 The example environment shown is 100.

[0039] As mentioned above, image processing technology is being applied to all aspects of people's lives. It provides diverse video and game scenes. Currently, high frame rates and high image quality remain core user demands. To eliminate screen tearing, Vertical Sync (VSync) technology is widely used. However, the rendering load of modern game scenes is highly dynamic, involving complex geometric calculations or texture processing, which often leads to fluctuations in single-frame rendering time, resulting in a decrease in image quality.

[0040] To improve smoothness with limited hardware performance, frame generation technology has emerged. Currently, there are two main types of frame generation technology: interpolation and extrapolation. While interpolated frames generated using interpolation are of higher quality, they heavily rely on future frames being rendered within a predetermined timeframe, resulting in inherent display latency. Extrapolation, on the other hand, predicts the next frame based solely on historical frames and motion vectors, thus lacking future information (such as subsequent frames). Consequently, extrapolation is prone to artifacts at screen edges or in occluded areas, failing to guarantee the quality of the interpolated frames.

[0041] To address the aforementioned problems, this disclosure provides an image processing method, with reference to... Figure 2 In step 210, the electronic device 110 or the server 130, in response to the rendering system being in a stable state, performs a first cycle of rendering according to a first rendering plan. The first rendering plan indicates that at least one first interpolated frame is displayed between a first start rendering frame and a first end rendering frame. The at least one first interpolated frame includes at least an interpolated frame generated based on the first end rendering frame.

[0042] As an example, such as Figure 3 As shown, Figure 3 Timing diagrams of image processing according to some embodiments of this disclosure are shown. Electronic device 110 or server 130 may utilize multiple modules for image processing. These multiple modules may include, for example, a rendering engine 310, a frame interpolation generator 311, a display queue 312, and a display. The frame interpolation generator 311 includes an interpolation unit and an extrapolation unit. The interpolation unit can be used to receive image data from two consecutive rendered frames and optical flow / motion vectors. The interpolation unit can synthesize intermediate time-stamp images using deep learning or optical flow algorithms. The interpolation unit supports asymmetric temporal weighting, allowing the generation of interpolated frames biased towards arbitrary time points. The extrapolation unit can perform state-based reprojection.

[0043] During image processing, electronic device 110 or server 130 can execute multiple rendering plans, which can correspond to multiple rendering cycles. These multiple rendering cycles can include a first rendering cycle, a second rendering cycle, a third rendering cycle, and a fourth rendering cycle. Each of these rendering cycles can correspond to a fixed display interval. For example, following vertical synchronization technology, image frames are submitted according to the display's refresh cycle. The first rendering cycle can, for example, be from time T-2 to time T0. The first rendering cycle includes a buffer window for two vertical synchronization cycles.

[0044] During the first rendering cycle, the rendering system of electronic device 110 or server 130 can be in a stable state. This stable state can, for example, indicate that the first end rendering frame has completed rendering within a preset duration. For example, the first end rendering frame that needs to be displayed at the second display time of the first end rendering frame has been rendered before the first display time of at least one first interpolated frame.

[0045] When the rendering system is in a stable state, electronic device 110 or server 130 can execute the first rendering plan. As an example, at 320, electronic device 110 or server 130 can retrieve the first initial rendering frame stored in the display queue 312. Further, electronic device 110 or server 130 can transmit the first initial rendering frame to the display so that the display can provide the user 313 with the image corresponding to the first initial rendering frame at time T-2. At 321, after the rendering engine 310 completes the rendering of the first final rendering frame within a preset time period (e.g., before the display time of the first interpolated frame), electronic device 110 or server 130 can provide the first final rendering frame to the interpolation generator 311 for generating a second interpolated frame. At 322, electronic device 110 or server 130 can also store the first final rendering frame in the display queue 312 to await output. Additionally, at 323, electronic device 110 or server 130 can also perform an interpolation operation based on the first initial rendering frame and the first final rendering frame using the interpolation generator 311 to generate a second interpolated frame. Since the second interpolated frame is output midway between the first start rendering frame and the first end rendering frame, it can be generated using a first time weight. This first time weight could be, for example, 1 / 2. After the second interpolated frame is generated, the electronic device 110 or server 130 can store it in the display queue 312 to await output. Further, the electronic device 110 or server 130 can output the second interpolated frame and the first end rendering frame sequentially according to the display's refresh rate. For example, at 324, the electronic device 110 or server 130 can output the first interpolated frame at time T-1. At 325, after the second interpolated frame is output, the electronic device 110 or server 130 can output the first end rendering frame at time T0.

[0046] In step 220, the electronic device 110 or the server 130, in response to the failure to complete rendering of the second end rendering frame at a preset time in the second rendering cycle, executes rendering of the second rendering cycle according to a second rendering plan. The second rendering plan indicates that at least one second interpolated frame is displayed between the second start rendering frame and the second end rendering frame. All at least one second interpolated frame is an extrapolated frame.

[0047] As an example, the electronic device 110 or server 130 determines whether the second end rendering frame has been rendered at a preset time. For example, whether the second end rendering frame has been rendered before the display time of the next interpolation frame. If the second end rendering frame has not been rendered at the preset time, the electronic device 110 or server 130 determines that the rendering pipeline is in an abnormal state, and then executes the rendering of the second rendering cycle with the second rendering plan. The second rendering cycle here can be, for example, from time T0 to time T2. Taking a racing game scene as an example, the car controlled by the player crashes into the guardrail, triggering complex particle effects. At this time, the computational load of the graphics processing unit surges, causing the second end rendering frame, which should have been output at time T1, to fail to be rendered on time. At this time, the electronic device 110 or server 130 can execute the second rendering plan to remedy the situation.

[0048] In some embodiments, the electronic device 110 or server 130 may display a second starting rendering frame at a first moment. The electronic device 110 or server 130 performs interpolation frame processing on the second starting rendering frame to generate a first interpolated frame. Further, the electronic device 110 or server 130 may display the first interpolated frame at a second moment and display a second ending rendering frame at a third moment.

[0049] As an example, electronic device 110 or server 130 performs rendering for the second rendering cycle according to the second rendering plan, such as... Figure 3As shown. At 325, the electronic device 110 or server 130 can display the second starting rendering frame on the display at a first moment (e.g., time T0). This second starting rendering frame can, for example, be the same rendering frame as the first ending rendering frame. Further, at 326, since the second ending rendering frame has not been fully rendered, to ensure smoothness of the image, the electronic device 110 or server 130 can perform extrapolation processing based on the second starting rendering frame using the interpolation generator 311 to generate a first extrapolated frame. After the first extrapolated frame is generated, the electronic device 110 or server 130 can store the first extrapolated frame in the display queue 312 to await output. At 327, the electronic device 110 or server 130 can retrieve the second interpolated frame (i.e., the first extrapolated frame) from the display queue 312 and provide the second interpolated frame to the display to output the first extrapolated frame at a second moment (e.g., time T1). At 328, rendering engine 310 completes rendering of the second end-rendered frame, and electronic device 110 or server 130 can provide the second end-rendered frame to interpolation generator 311 for generating subsequent interpolated frames. At 329, electronic device 110 or server 130 can store the second end-rendered frame in display queue 312 to await output. At 330, electronic device 110 or server 130 can retrieve the second end-rendered frame from display queue 312 at a third time (e.g., time T2) and provide it to the display for display.

[0050] While the second final rendering frame is not yet rendered, embodiments of this disclosure can generate and display a second interpolated frame, thus ensuring that the video output port can still respond to the vertical synchronization signal on time even when the rendering pipeline is blocked. In this way, embodiments of this disclosure can avoid screen pauses or repeated display of historical frames, thereby effectively improving the smoothness of the video.

[0051] In step 230, after the second rendering cycle ends, the electronic device 110 or the server 130 executes the rendering of the third rendering cycle according to the third rendering plan. The third rendering plan indicates that the number of interpolation frames between the third start rendering frame and the third end rendering frame is increased so that the duration of the third rendering cycle is longer than that of the first and second rendering cycles.

[0052] As an example, after the second rendering cycle ends, in order to restore a stable state, electronic device 110 or server 130 can generate multiple interpolated frames in the third rendering cycle (e.g., T2 to T5). Furthermore, electronic device 110 or server 130 can sequentially output multiple interpolated frames before the third end rendering frame, according to the display's refresh cycle, to delay the output time of the third end rendering frame. By delaying the output time of the third rendering frame, the output order of subsequent image frames can be guaranteed to be the start rendering frame - interpolated frame - end rendering frame, thereby restoring the stable state of the rendering pipeline.

[0053] In some embodiments, the electronic device 110 or server 130 may display a third starting rendering frame at a fourth time. The electronic device 110 or server 130 may display a second extrapolated frame at a fifth time. The second extrapolated frame is generated by performing extrapolation frame processing on the third starting rendering frame. At a sixth time, the electronic device 110 or server 130 may display a first interpolated frame. The first interpolated frame is generated by performing interpolation frame processing on the third starting rendering frame and the third ending rendering frame. Further, the electronic device 110 or server 130 may display a third ending rendering frame at a seventh time.

[0054] As an example, such as Figure 3 As shown, the third rendering cycle can be, for example, from time T2 to time T5. The third rendering cycle includes a long buffer window with three vertical synchronization cycles. Within this window, the electronic device 110 or the server 130 can employ a hybrid generation strategy to maintain visual coherence.

[0055] At time 330, electronic device 110 or server 130 may retrieve the third starting rendering frame from display queue 312 at a fourth time (e.g., time T2). Electronic device 110 or server 130 may provide the third starting rendering frame to the display so that the image of the third starting rendering frame can be presented to user 313 via the display. Here, the third starting rendering frame may, for example, be the same rendering frame as the second ending rendering frame. At time 331, before the fifth time (e.g., time T3), since the third ending rendering frame has not been rendered, electronic device 110 or server 130 may perform an extrapolation operation based on the third starting rendering frame through the extrapolation unit in frame interpolation generator 311 to generate a second extrapolated frame. After the second extrapolated frame is generated, electronic device 110 or server 130 may store the second extrapolated frame in display queue 312. At time 332, electronic device 110 or server 130 may retrieve the second extrapolated frame from display queue 312 at the fifth time (e.g., time T3) and provide it to the display. After receiving the second extrapolated frame, the display may display the second extrapolated frame. At 333, rendering engine 310 completes rendering of the third end rendering frame. Electronic device 110 or server 130 can provide the third end rendering frame to frame interpolation generator 311 for generating a first interpolated frame. At 334, electronic device 110 or server 130 can store the third end rendering frame in display queue 312 to await output. At 335, electronic device 110 or server 130 can utilize the interpolation unit in frame interpolation generator 311 to perform interpolation operations based on the third start rendering frame and the third end rendering frame to generate the first interpolated frame. After the first interpolated frame is generated, electronic device 110 or server 130 can store the first interpolated frame in display queue 312. Further, at 336, electronic device 110 or server 130 can retrieve the first interpolated frame from display queue 312 at a sixth time (e.g., time T4) and output it to the display. At 337, after the first interpolated frame is displayed, the electronic device 110 or the server 130 may, at the seventh moment (e.g., T5), retrieve the third end-rendered frame from the display queue 312 and output it to the display.

[0056] The embodiments of this disclosure can delay the output time of the third end rendering frame and output multiple interpolated frames before the output time of the third end rendering frame, thereby utilizing additional time slots to absorb jitter delays in the rendering pipeline and allowing the rendering pipeline to return to a stable state. In this way, the quality of image processing can be effectively improved.

[0057] In some embodiments, since two interpolated frames are output in the third rendering cycle, and the first interpolated frame is output after the second extrapolated frame, the electronic device 110 or server 130 can use asymmetric weights to generate the first interpolated frame. The asymmetric weights here could be, for example, 2 / 3.

[0058] In some embodiments, after the third rendering cycle has ended, rendering for the fourth rendering cycle is performed using the first rendering plan. As an example, after the rendering pipeline has returned to a stable state, the electronic device 110 or server 130 may perform rendering using the first rendering plan. Figure 3 As shown, at 337, the electronic device 110 or server 130 can retrieve the fourth starting rendering frame from the display queue 312 and output the fourth starting rendering frame at time T5. This fourth starting rendering frame can, for example, be the same rendering frame as the third ending rendering frame. At 338, after the rendering engine 310 completes the fourth ending rendering frame, it can provide the fourth ending rendering frame to the frame interpolation generator 311. At 339, the electronic device 110 or server 130 can store the fourth ending rendering frame in the display queue 312. At 340, the electronic device 110 or server 130 can perform an interpolation operation based on the fourth starting rendering frame and the fourth ending rendering frame using the frame interpolation generator 311 to generate a third interpolated frame. At 341, the electronic device 110 or server 130 can retrieve the third interpolated frame from the display queue 312 and output the third interpolated frame at time T6. At 342, the electronic device 110 or server 130 can output the fourth ending rendering frame at time T7 to complete the first rendering plan.

[0059] In this way, after an anomaly occurs in the rendering pipeline, the embodiments of this disclosure can restore the rendering pipeline to a stable state by executing a third rendering plan. Once the rendering pipeline has recovered to a stable state, the embodiments of this disclosure can continue executing the first rendering plan, thereby ensuring that interpolation is used to generate interpolated frames for the majority of the overall rendering process. Thus, the embodiments of this disclosure can maintain high image quality for most of the time and eliminate performance bottlenecks in extreme cases, greatly improving the user's overall visual experience.

[0060] In some embodiments, in a first rendering plan, electronic device 110 or server 130 may generate multiple interpolated frames. Specifically, the number of at least one first interpolated frame in the first rendering plan is determined based on the usage status of rendering resources. As an example, in a first rendering plan, electronic device 110 or server 130 may determine the usage status of rendering resources. The usage status of rendering resources may, for example, indicate the load trend of a graphics processing unit (GPU). This load trend may, for example, be determined by an appropriate algorithm (e.g., a predictive model). For example, electronic device 110 or server 130 may predict the load trend of the GPU based on statistical characteristics (e.g., weighted moving average and variance) of historical rendering time data of the GPU using a predictive model.

[0061] If the usage status of the rendering resources meets a first condition (e.g., the load trend reaches a first threshold), the electronic device 110 or server 130 can increase the number of first interpolated frames generated in the first rendering plan. For example, the electronic device 110 or server 130 can first generate an extrapolated frame A using the first starting rendering frame in the first rendering plan. Further, the electronic device 110 or server 130 can use the first starting rendering frame and the first ending rendering frame to generate an interpolated frame B to improve the smoothness of the image. Here, the interpolated frame B can be generated, for example, using asymmetric weights. This method of determining the number of interpolated frames can be applied not only to the first rendering plan but also to subsequent rendering plans (e.g., the second rendering plan, the third rendering plan, etc.).

[0062] For example, in the second rendering plan, electronic device 110 or server 130 can generate multiple extrapolated frames. Specifically, electronic device 110 or server 130 can generate extrapolated frame C based at least on the second initial rendering frame. Further, electronic device 110 or server 130 can also generate extrapolated frame D based on the second initial rendering frame and extrapolated frame C.

[0063] For example, in the third rendering plan, electronic device 110 or server 130 can generate an extrapolated frame E based at least on the third starting rendering frame. Further, electronic device 110 or server 130 can generate an extrapolated frame F based on the third starting rendering frame and the extrapolated frame E. After the above two extrapolated frames are generated, electronic device 110 or server 130 can also use asymmetric weights to generate an interpolated frame G based on the third starting rendering frame and the third ending rendering frame.

[0064] In some scenarios, if multiple extrapolated frames were generated in the previous rendering cycle, the electronic device 110 or server 130 may not need to postpone the end rendering frame for several moments in the current rendering cycle. In the current rendering cycle, the electronic device 110 or server 130 may generate only one interpolated frame. If at least one extrapolated frame still needs to be generated in the next rendering cycle, then the electronic device 110 or server 130 may execute a third rendering plan in the next rendering cycle.

[0065] If the usage status of the rendering resources meets the second condition (e.g., the load trend is less than the first threshold), the electronic device 110 or the server 130 can reduce the number of first interpolated frames generated in the first rendering plan. For example, no interpolated frames are generated in the first rendering plan. This method of determining the number of interpolated frames can be applied not only to the first rendering plan but also to subsequent rendering plans (e.g., the second rendering plan, the third rendering plan, etc.).

[0066] In this way, the embodiments of this disclosure can prevent frequent changes to the rendering schedule, thereby increasing the stability of the rendering pipeline and maintaining a stable output frame rate even in unstable rendering environments, while maximizing the visual quality of each frame.

[0067] Through the above-described solutions, the embodiments of this disclosure enhance the robustness of the real-time rendering system in a vertically synchronized environment, eliminating screen stuttering caused by rendering time fluctuations. By switching between interpolation and extrapolation modes, an interpolated frame can be generated based on the current frame, filling in the gaps caused by rendering delays. This avoids screen freezing or repeated display due to dropped frames, ensuring visual continuity. Furthermore, the embodiments of this disclosure can restore the stability of the rendering pipeline by executing a third rendering plan. This allows the display rhythm of the vertically synchronized environment to be met without discarding any critical rendering tasks, and the frame queue required for interpolation to be refilled, achieving a smoother transition of the rendering pipeline from an abnormal state to a stable state.

[0068] Example devices and equipment

[0069] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural diagram of an example device 400 for graphics processing is shown, according to some scenarios. Device 400 may be implemented as or included in electronic device 110 or server 130. The various modules / components in device 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0070] like Figure 4 As shown, the apparatus 400 includes: a first execution module 410 configured to execute rendering of a first rendering cycle according to a first rendering plan in response to the rendering system being in a stable state, the first rendering plan indicating that at least one first interpolated frame is displayed between a first start rendering frame and a first end rendering frame, the at least one first interpolated frame including at least an interpolated frame generated based on the first end rendering frame; a second execution module 420 configured to execute rendering of a second rendering cycle according to a second rendering plan in response to the second end rendering frame not being rendered at a preset time in the second rendering cycle, the second rendering plan indicating that at least one second interpolated frame is displayed between the second start rendering frame and the second end rendering frame, the at least one second interpolated frame being an extrapolated frame; and a third execution module 430 configured to execute rendering of a third rendering cycle according to a third rendering plan after the second rendering cycle ends, the third rendering plan indicating that the number of interpolated frames between the third start rendering frame and the third end rendering frame is increased such that the duration of the third rendering cycle is longer than that of the first rendering cycle and the second rendering cycle.

[0071] In some embodiments, the second execution module 420 is further configured to: display a second starting rendering frame at a first moment; perform interpolation frame processing on the second starting rendering frame to generate a first interpolation frame; and display the first interpolation frame at a second moment and display a second ending rendering frame at a third moment.

[0072] In some embodiments, the third execution module 430 is further configured to: display a third start rendering frame at a fourth time; display a second extrapolated frame at a fifth time, the second extrapolated frame being generated by performing extrapolation frame processing on the third start rendering frame; display a first interpolated frame at a sixth time, the first interpolated frame being generated by performing interpolation frame processing on the third start rendering frame and the third end rendering frame; and display a third end rendering frame at a seventh time.

[0073] In some embodiments, the apparatus 400 further includes a fourth execution module configured to perform rendering of the fourth rendering cycle according to the first rendering schedule after the third rendering cycle has ended.

[0074] In some embodiments, each of the first, second, and third rendering cycles corresponds to a fixed display interval.

[0075] In some embodiments, the number of at least one first interpolation frame in the first rendering plan is determined based on the usage status of the rendering resources.

[0076] In some embodiments, the apparatus 400 further includes an adjustment module configured to increase the number of rendering resources in response to a first condition being met, and to decrease the number of rendering resources in response to a second condition being met.

[0077] Figure 5 A block diagram of a computing device 500 in which various embodiments of the present disclosure may be implemented is shown. The computing device 500 may be implemented as an electronic device 110 or a server 130, or may be included in an electronic device 110 or a server 130.

[0078] It should be understood that, Figure 5 The computing device 500 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0079] like Figure 5 As shown, computing device 500 includes general-purpose computing device 500. Computing device 500 may include at least one or more processors or processing units 510, memory 520, storage unit 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560.

[0080] In some embodiments, the computing device 500 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, a large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 500 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0081] Processing unit 510 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 500. Processing unit 510 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0082] Computing device 500 typically includes various computer storage media. Such media can be any media accessible by computing device 500, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 530 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 500.

[0083] The computing device 500 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 5 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0084] Communication unit 540 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in computing device 500 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, computing device 500 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0085] Input device 550 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 560 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 540, computing device 500 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 500 can also communicate with one or more devices that enable a user to interact with computing device 500, or, if needed, with any device that enables computing device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via an input / output (I / O) interface (not shown).

[0086] In some embodiments, some or all of the components of computing device 500 may be deployed in a cloud computing architecture, rather than being integrated into a single device. In a cloud computing architecture, components may be remotely provided and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (WAN), such as the Internet, using suitable protocols. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across remote data center locations. Cloud computing infrastructure may provide services through shared data centers, although to users they appear as a single access point. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from service providers at remote locations. Alternatively, the components and functionality described herein may be provided by conventional servers or installed directly or otherwise on client devices.

[0087] In embodiments of this disclosure, computing device 500 may be used to implement image processing. Memory 520 may include one or more image processing modules 525 having one or more program instructions. These modules are accessible and executable by processing unit 510 to perform the functions of the various embodiments described herein.

[0088] In an example embodiment of image processing, the virtual scene may be processed, for example, by the image processing module 525 to generate a picture of the virtual scene or a bitstream corresponding to the picture of the virtual scene. The picture or bitstream of the virtual scene may be provided as output 570 via output device 560.

[0089] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. A method of image processing, characterized by, The method comprises: in response to the rendering system being in a stable state, performing rendering of a first rendering cycle with a first rendering plan, the first rendering plan indicating displaying at least one first interpolation frame between a first start rendering frame and a first end rendering frame, the at least one first interpolation frame comprising at least an interpolation frame generated based on the first end rendering frame; in response to the rendering of a second end rendering frame not being completed at a preset time of a second rendering cycle, performing rendering of the second rendering cycle with a second rendering plan, the second rendering plan indicating displaying at least one second interpolation frame between a second start rendering frame and the second end rendering frame, the at least one second interpolation frame being all extrapolation frames; and after the second rendering cycle ends, performing rendering of a third rendering cycle with a third rendering plan, the third rendering plan indicating increasing the number of interpolation frames between a third start rendering frame and a third end rendering frame, so that the time length of the third rendering cycle is longer than that of the first rendering cycle and the second rendering cycle.

2. The method of claim 1, wherein, Performing rendering of the second rendering cycle with the second rendering plan comprises: at a first time, displaying the second start rendering frame; performing extrapolation frame processing on the second start rendering frame to generate a first extrapolation frame; and at a second time, displaying the first extrapolation frame, and at a third time, displaying the second end rendering frame.

3. The method of claim 1, wherein, After the second rendering cycle ends, performing rendering of a third rendering cycle with a third rendering plan comprises: at a fourth time, displaying the third start rendering frame; at a fifth time, displaying a second extrapolation frame, the second extrapolation frame being generated by performing extrapolation frame processing on the third start rendering frame; at a sixth time, displaying a first interpolation frame, the first interpolation frame being generated by performing interpolation frame processing on the third start rendering frame and the third end rendering frame; and at a seventh time, displaying the third end rendering frame.

4. The method of claim 1, wherein, The method further comprises: after the third rendering cycle ends, performing rendering of a fourth rendering cycle with the first rendering plan.

5. The method of claim 1, wherein, Each of the first rendering cycle, the second rendering cycle, and the third rendering cycle corresponds to a fixed display interval.

6. The method of claim 1, wherein, The number of the at least one first interpolation frame in the first rendering plan is determined based on a usage state of a rendering resource.

7. The method of claim 6, wherein, The method further comprises: in response to the usage state of the rendering resource satisfying a first condition, increasing the number; and in response to the usage state of the rendering resource satisfying a second condition, decreasing the number.

8. An apparatus for image processing, characterized by, The apparatus comprises: a first performing module configured to, in response to a rendering system being in a stable state, perform rendering of a first rendering cycle with a first rendering plan, the first rendering plan indicating displaying at least one first interpolation frame between a first start rendering frame and a first end rendering frame, the at least one first interpolation frame comprising at least an interpolation frame generated based on the first end rendering frame; a second execution module configured to, in response to the rendering of the second end rendering frame not being completed at the preset time in the second rendering period, perform rendering of the second rendering period according to a second rendering plan, the second rendering plan indicating that at least one second interpolation frame is displayed between a second start rendering frame and the second end rendering frame, the at least one second interpolation frame each being an extrapolation frame; and a third execution module configured to, after the second rendering period ends, perform rendering of a third rendering period according to a third rendering plan, the third rendering plan indicating that a number of interpolation frames between a third start rendering frame and a third end rendering frame is increased, so that a length of time of the third rendering period is longer than the first rendering period and the second rendering period.

9. An electronic device, comprising: The device includes: at least one processor; and at least one memory that is coupled to the at least one processor and stores instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the electronic device to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium has stored thereon computer-executable instructions that are executable by a processor to implement the method according to any one of claims 1 to 7.

11. A computer program product, characterised in that, The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 7.

Images