Video generation methods, devices, and media for the Android operating system

Abstract

This application provides a video generation method, device, and medium for the Android operating system. The method includes the following steps: In the Vulkan swap chain initialization phase: creating a main rendering thread and a dedicated display thread; and creating an Empty Semaphore and a Full Semaphore; In the runtime phase: the main rendering thread obtains an idle frame buffer through the swap chain and executes the rendering process of the current frame; the display thread monitors the Empty Semaphore state in real time, and when a non-empty signal is detected, submits the frame data to Surface Flinger for display. Specifically, after the main rendering thread completes the rendering of the current frame, it directly checks the Full Semaphore state; if it is not full, it obtains an idle frame buffer and executes the rendering process of the next frame. The method of this application can effectively reduce the latency caused by frame interpolation without modifying the underlying Android driver.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, specifically to a video generation method for the Android operating system. Background Technology

[0002] Video frame rate is a core indicator determining the visual smoothness of a game. High frame rate games provide users with smoother operation feedback and a more immersive visual experience. Therefore, improving game frame rates has become an important direction for technological development in the industry. With the advancement of display technology, the refresh rates of mainstream Android mobile devices have generally reached 60Hz, 120Hz, or even higher. However, the rate of improvement in game hardware performance cannot fully keep up with the high refresh rate requirements of display devices. Against this backdrop, video frame interpolation technology has emerged and is widely used in the gaming field.

[0003] Several game frame interpolation technologies have emerged, such as AMD's FidelityFX™ SuperResolution 4 (FSR 4) technology, NVIDIA's DLSS 4 Technology, and the FGSR (Fast and Lightweight Super Resolution) technology optimized for mobile devices proposed in academia. These frame interpolation technologies generate intermediate frames between the original rendered frames of the game, effectively increasing the output frame rate of the game even when the graphics card performance is insufficient. This allows low-performance devices to present visual effects close to high refresh rates, significantly lowering the hardware requirements for gaming and improving the user experience.

[0004] However, under the Vulkan rendering architecture of Android, the application of traditional frame interpolation technology faces serious latency issues, primarily stemming from inherent flaws in the Vulkan swap chain. In the existing Vulkan rendering pipeline, display operations and rendering operations share main thread resources. Display operations can block the entire game's rendering pipeline, meaning the rendering pipeline can only begin rendering the next frame after the previous frame has finished displaying. When frame interpolation is introduced, the processing flow of each frame adds the generation and display of intermediate frames. This further amplifies the blocking effect of display operations, leading to a significant reduction in the overall efficiency of the rendering pipeline. Not only does it fail to fully leverage the frame rate improvement advantages of frame interpolation, but it can also cause negative impacts such as frame rate fluctuations and operational latency.

[0005] Therefore, there is a need to provide a video generation method that can eliminate the delay caused by frame interpolation. Summary of the Invention

[0006] In view of this, this application provides a video generation method, device and medium for the Android operating system to solve the above-mentioned technical problems in the prior art.

[0007] According to one aspect of this application, a video generation method for an Android operating system is provided, the method comprising the following steps:

[0008] During the Vulkan swap chain initialization phase:

[0009] Create a main rendering thread and a dedicated display thread;

[0010] It also creates an Empty Semaphore and a Full Semaphore, with the Empty Semaphore initialized to empty and the Full Semaphore initialized to full.

[0011] During operation:

[0012] The main rendering thread obtains an idle frame buffer through the swap chain, executes the rendering process of the current frame, and submits the frame data to the GPU through the vkQueueSubmit interface after rendering is complete. At the same time, it sends a non-empty signal to the Empty Semaphore to indicate that there is currently displayable frame data.

[0013] The display thread monitors the Empty Semaphore state in real time. When a non-empty signal is detected, it submits the frame data to the Surface Flinger for display via the vkQueuePresentKHR interface. After the display operation is completed, it sends a non-full signal to the FullSemaphore to indicate that the frame buffer has been released.

[0014] When the main rendering thread finishes rendering the current frame, it directly checks the Full Semaphore status. If it is not full, it obtains an idle frame buffer and executes the rendering process for the next frame.

[0015] According to a preferred embodiment of this application, the rendering process includes rendering the original frame and rendering the inserted intermediate frame.

[0016] According to a preferred embodiment of this application, the frame rate of the original frame is 30fps, and the frame rate of the intermediate frame is 30fps.

[0017] According to a preferred embodiment of this application, the method is used to generate videos in a game engine running on an Android terminal device.

[0018] According to a preferred embodiment of this application, the method is used to generate videos in a game engine running on an Android game server.

[0019] According to a preferred embodiment of this application, the method further includes allocating an independent command pool for the display thread through vkCreateCommandPool during the Vulkan exchange chain initialization phase.

[0020] According to a preferred embodiment of this application, the method further includes, during the runtime phase, monitoring the execution status of rendering and display operations through the Vulkan vkGetFenceStatus interface; if a timeout or error occurs, triggering a semaphore reset and queue restart process.

[0021] According to a preferred embodiment of this application, the intermediate frame is generated using FSR 4, DLSS 4, or FGSR frame interpolation technology.

[0022] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0023] One or more processors;

[0024] Memory, which is used to store executable instructions;

[0025] The one or more processors are configured to implement the methods described above via executable instructions.

[0026] According to another aspect of this application, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, causes the processor to perform the method described above.

[0027] The technical solution of this application implements asynchronous display control, synchronous mechanism reuse, asynchronous trigger rendering, and idle queue allocation, which can effectively reduce the latency caused by frame interpolation without modifying the underlying Android driver.

[0028] Asynchronous display control: By utilizing unused queues in the Vulkan queue family that are not currently being rendered, a dedicated display thread is created, completely separating the display operations (present) that were originally integrated into the main rendering thread. This display thread is specifically responsible for interacting with the Android surface flinger and performing frame submission and display operations, allowing the main rendering thread to focus on game rendering and frame interpolation calculations, thus avoiding the blocking of the rendering process by display operations.

[0029] Synchronization Mechanism Reuse: The Vulkan standard semaphore synchronization mechanism is reused to build a bidirectional synchronization system of "Empty Semaphore - Full Semaphore," eliminating the need for custom synchronization logic. State interaction between the main rendering thread and the dedicated display thread is achieved through standard synchronization interfaces, ensuring orderly connection between frame data generation and display processes while avoiding resource contention between threads.

[0030] Asynchronous Triggered Rendering: The startup logic of the rendering pipeline has been restructured. After the main rendering thread completes the rendering and data submission of the current frame (including the original frame and the intermediate frames generated by interpolation), it can directly start the rendering process of the next frame without waiting for the display thread to return a signal indicating that the previous frame has been displayed. By eliminating the serial dependency between rendering and display, pipelined execution of rendering operations is achieved, significantly improving pipeline utilization.

[0031] Idle queue allocation: This module dynamically detects and allocates Vulkan queue resources, ensuring that queues allocated to the display thread are used solely for display operations and do not participate in rendering calculations or data caching. It queries the queue family type and occupancy status via the Vulkan interface, prioritizing queues with display capabilities that are currently idle to avoid additional cache resource consumption and reduce memory overhead. Attached Figure Description

[0032] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application, but do not constitute a limitation on the technical solutions of this application.

[0033] Figure 1 A timing diagram of video generation in the prior art is shown;

[0034] Figure 2 An exemplary timing diagram of the method provided by an exemplary embodiment of this application is shown;

[0035] Figure 3 A schematic diagram illustrating the synchronization process between the display thread and the rendering thread in an exemplary embodiment of this application is shown;

[0036] Figure 4 A schematic diagram of a frame buffer in an exemplary embodiment of this application is shown;

[0037] Figure 5 A flowchart illustrating timeout reset in an exemplary embodiment of this application is shown;

[0038] Figure 6 A structural block diagram of an electronic device provided by an exemplary embodiment of this application is shown. Detailed Implementation

[0039] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. The descriptions of the exemplary embodiments are merely illustrative and are not intended to limit the scope of this application or its application or use. This application can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to make the application thorough and complete, and to fully express the scope of this application to those skilled in the art.

[0040] Unless explicitly stated otherwise, an element may be one or more unless otherwise specified. The terms “multiple / several” mean two or more, the term “based on” should be interpreted as “at least partially based on,” and the terms “and / or” and “at least one of…” cover any one of the listed items and all possible combinations thereof. Furthermore, expressions such as “first,” “second,” etc., are for descriptive purposes only and do not indicate or imply their relative importance or implicitly specify the number of technical features indicated.

[0041] Figure 1 The diagram illustrates a timing sequence for video generation in the prior art. In the Vulkan rendering architecture of the Android system, display and rendering operations share main thread resources; the rendering pipeline can only begin the rendering process for the next frame after the previous frame has been displayed. For example... Figure 1 As shown, the rendering pipeline first renders the original frame 1, and then renders the inserted intermediate frame 0.5. After rendering the original frame 1 and intermediate frame 0.5, intermediate frame 0.5 is displayed first. At this time, because the display operation and the rendering operation share the main thread resources, the rendering pipeline does not start rendering the original frame 2, but must wait for the original frame 1 to finish displaying before starting the rendering process of the original frame 2. Therefore, during the time interval between displaying intermediate frame 0.5 and displaying the original frame 1, the hardware for rendering is idle. After the original frame 1 is displayed, the rendering pipeline starts the rendering process of the original frame 2 and the inserted intermediate frame 1.5 in sequence. Only after the rendering process of the inserted intermediate frame 1.5 is completed can the rendered frame 1.5 be displayed, thus introducing a large delay between the display of intermediate frame 1.5 and the display of the original frame 1. It can be seen that when performing frame interpolation operations under the Vulkan rendering architecture of the Android system, not only are hardware resources not fully utilized, but a large delay is also generated, affecting the user experience.

[0042] The method implemented in this application separates the display operations that were originally integrated into the main rendering thread. The display thread is specifically responsible for interacting with the Android Surface Flinger and performing frame submission and display operations, allowing the main rendering thread to focus on game screen rendering and frame interpolation calculations, thus avoiding the blocking of the rendering process by display operations. Specifically, the video generation method for the Android operating system according to the exemplary embodiment of this application includes the following steps:

[0043] 1. During the Vulkan switch chain initialization phase:

[0044] - Create a main rendering thread and a dedicated display thread. The main rendering thread is already present in the Vulkan rendering architecture. The dedicated display thread is independent of the main rendering thread, thus preventing the display of a frame from hindering the rendering process of subsequent frames. A separate command pool can be allocated to the display thread using vkCreateCommandPool to ensure isolation of command resources between threads.

[0045] - Create an Empty Semaphore and a Full Semaphore. The Empty Semaphore is initialized to empty; the Full Semaphore is initialized to full.

[0046] The display thread and the rendering thread are synchronized directly using the standard producer-consumer model. Figure 3 This illustrates the synchronization process between the display thread and the rendering thread. During initialization, the full semaphore is in a full state, and the empty semaphore is in an empty state.

[0047] 2. During the operational phase:

[0048] The main rendering thread acquires an idle frame buffer through the swap chain, executes the rendering process for the current frame, and submits the frame data to the GPU via the vkQueueSubmit interface after rendering is complete. Simultaneously, it sends a non-empty signal to the Empty Semaphore to indicate that there is currently displayable frame data. The rendering process includes rendering the original frame and rendering the inserted intermediate frames. Intermediate frames can be generated using known frame interpolation algorithms such as FSR 4, DLSS 4, or FGSR. Figure 4 A schematic diagram of an idle frame buffer in an exemplary embodiment of this application is shown. Figure 4 In this architecture, rendering and display share three VkImage buffers. During rendering, one buffer is set as the target. During display, the content of the corresponding buffer is displayed on the display device. Buffers that are neither in rendering nor display use become idle Images.

[0049] - The display thread monitors the Empty Semaphore state in real time. When a non-empty signal is detected, the frame data is submitted to Surface Flinger for display through the vkQueuePresentKHR interface. After the display operation is completed, a non-full signal is sent to FullSemaphore to indicate that the frame buffer has been released.

[0050] During the runtime phase, after the main rendering thread finishes rendering the current frame, it directly checks the Full Semaphore state. If it is not full, it obtains an idle frame buffer and executes the rendering process of the next frame. Unlike existing technologies, it does not need to wait for the current frame to be displayed before starting the rendering process of the next frame, thus achieving parallel execution of rendering and display.

[0051] To handle frame data anomalies or thread state errors, the method according to embodiments of this application may further include an anomaly detection step. This involves monitoring the execution status of rendering and display operations via Vulkan's vkGetFenceStatus interface. If a timeout or error occurs, a semaphore reset and queue restart process (i.e., a timeout reset process) is immediately triggered to ensure the pipeline is not blocked and to guarantee the stability of the method's operation. For details on the timeout reset process, please refer to [link to relevant documentation]. Figure 5 .

[0052] The method according to the embodiments of this application can be used to generate videos in a game engine running on an Android terminal device, or in a game engine running on an Android game server.

[0053] Figure 2 An exemplary timing diagram of the method provided by an exemplary embodiment of this application is shown. The method according to the exemplary embodiment of this application creates a main rendering thread and a separate dedicated display thread, thus enabling the rendering process and the display process to be executed independently.

[0054] like Figure 2 As shown, after the main rendering thread completes the rendering of the original frame 1 and the inserted intermediate frame 0.5, the dedicated display thread sequentially executes the display of intermediate frame 0.5 and the original frame 1. Simultaneously, the main rendering thread does not need to wait for the original frame 1 to finish displaying before starting the rendering process for the original frame 2; instead, it starts the rendering process for the original frame 2 directly after the intermediate frame 0.5 is rendered. Therefore, the main rendering thread can continuously and uninterruptedly render the original frame and the inserted intermediate frame, while the dedicated display thread can display the original frame and the inserted intermediate frame at a fixed frame rate, avoiding the latency issues present in existing technologies.

[0055] Figure 2In the exemplary timing diagram shown, the frame rate of the original frame can be, for example, 30 fps. By inserting an intermediate frame (also with a frame rate of 30 fps) between two consecutive original frames, the total frame rate of the resulting video can be stacked to 60 fps.

[0056] In addition to the above-described technical solutions, the present invention also provides an electronic device running the Android operating system, the electronic device including one or more processors and a memory for storing executable instructions. The one or more processors are configured to implement the above-described methods via executable instructions. The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to perform the above-described methods. In the following sections of this specification, [further details will be provided]. Figure 3 Illustrative examples are provided to describe the aforementioned electronic devices and computer-readable storage media.

[0057] Figure 6 An example configuration of an electronic device 300 that can be used to implement the methods described in this invention is shown. The technical solutions of this invention can also be implemented wholly or at least partially by the electronic device 300 or similar devices / systems. The electronic device 300 can be of various different types. Examples of the electronic device 300 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices, wearable devices, entertainment devices, televisions or other display devices, automotive computers, etc.

[0058] Electronic device 300 may include at least one processor 302, memory 304, multiple communication interfaces 309, display device 301, other input / output (I / O) devices 310, and one or more mass storage devices 303 capable of communicating with each other via system bus 311 or other suitable connection.

[0059] Processor 302 may be one or more processing units, and all processing units may include one or more computing units or multiple cores. Processor 302 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 302 may be configured to fetch and execute computer-readable instructions stored in memory 304, mass storage device 303, or other computer-readable media, such as program code of operating system 305, application program 306, or other program 307.

[0060] Memory 304 and mass storage device 303 are examples of computer-readable storage media for storing instructions executed by processor 302 to perform the various functions described above. For example, memory 304 can generally include both volatile and non-volatile memory. Furthermore, mass storage device 303 can generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks, storage arrays, network-attached storage, storage area networks, etc. Memory 304 and mass storage device 303 can both be collectively referred to as memory or computer-readable storage media in this invention, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which can be executed by processor 302 as a specific machine configured to perform the operations and functions described in the examples of this invention.

[0061] Multiple programs can be stored on mass storage device 303. These programs include operating system 305, one or more application programs 306, other programs 307, and program data 308, and they can be loaded into memory 304 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: the methods provided by the present invention (including any suitable steps of the method) and / or other embodiments described in the present invention.

[0062] Although Figure 6 The data is shown as stored in the memory 304 of the electronic device 300, but the modular operating system 305, application program 306, other programs 307, and program data 308, or portions thereof, can be implemented using any form of computer-readable medium accessible by the electronic device 300. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer. A communication medium includes a medium through which communication signals, such as computer-readable instructions, data structures, program modules, or other data, are transmitted from one system to another. The communication medium can include guided transmission media and wireless media capable of propagating energy waves. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals, for example, in a wireless medium.

[0063] For example, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other known media or those developed hereafter capable of storing computer-readable information / data for use by a computer system.

[0064] One or more communication interfaces 309 are used to exchange data with other devices via means such as a network or direct connection. This communication interface can be one or more of the following: any type of network interface, wired or wireless (such as WLAN) interface, Wi-MAX interface, Ethernet interface, USB interface, cellular network interface, Bluetooth interface, NFC interface, etc. Communication interface 309 can facilitate communication across various network and protocol types, including wired and wireless networks, the Internet, etc. Communication interface 309 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0065] In some examples, a display device 301, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 310 may be devices that receive user input and provide output to the user, and may include touch / gesture input devices, cameras, keyboards, remote controls, mice, audio input / output devices, etc.

[0066] The technical solutions described in this invention can be supported by various configurations of the electronic device 300, and are not limited to the specific examples of the technical solutions described in this invention. The foregoing description and illustrations of this invention are not restrictive. It will be apparent to those skilled in the art that this invention is not limited to the details of the exemplary embodiments described above, and that this invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the scope of protection claimed by this invention is defined by the claims rather than the foregoing description, and all variations falling within the meaning and scope of equivalents of the claims are covered within the scope of protection of this invention.

Claims

1. A method for video generation for an Android operating system, characterized in that, The method comprises the following steps: In the Vulkan swap chain initialization phase: A rendering main thread and a display dedicated thread are created; An Empty Semaphore and a Full Semaphore are also created, with the initial value of the Empty Semaphore being empty and the initial value of the Full Semaphore being full; In the running phase: The rendering main thread acquires an idle frame buffer through the swap chain, executes the rendering process of the current frame, and submits the frame data to the GPU through the vkQueueSubmit interface after the rendering is completed, and sends a non-empty signal to the Empty Semaphore to indicate that there is displayable frame data; The display thread listens to the state of the Empty Semaphore in real time, and when a non-empty signal is detected, it submits the frame data to the Surface Flinger for display through the vkQueuePresentKHR interface, and sends a non-full signal to the Full Semaphore after the display operation is completed, indicating that the frame buffer has been released, Wherein, after the rendering main thread completes the rendering of the current frame, it directly detects the state of the Full Semaphore, and if it is not full, it acquires an idle frame buffer and executes the rendering process of the next frame.

2. The method of claim 1, wherein, The rendering process includes the rendering of the original frame and the inserted intermediate frame.

3. The method of claim 2, wherein, The frame rate of the original frame is 30fps, and the frame rate of the intermediate frame is 30fps.

4. The method of claim 1, wherein, The method is used to generate video in a game engine running on an Android terminal device.

5. The method of claim 1, wherein, The method is used to generate video in a game engine running on an Android game server.

6. The method of claim 1, wherein, The method further comprises, in the Vulkan swap chain initialization phase, allocating an independent command pool for the display thread through vkCreateCommandPool.

7. The method of claim 1, wherein, The method further comprises, in the running phase, listening to the execution state of the rendering and display operations through the vkGetFenceStatus interface of Vulkan, and if there is a timeout or error, triggering the semaphore reset and queue restart process.

8. The method of claim 2, wherein, The intermediate frame is generated using FSR 4, DLSS 4 or FGSR frame insertion technology.

9. An electronic device, comprising: The electronic device comprises: One or more processors; Memory for storing executable instructions; The one or more processors are configured to implement the method of any one of claims 1 to 8 via the executable instructions.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 8.

Images