Real-time rendering method for accessing AI super-division algorithm in Android vehicle-mounted display system

Abstract

The invention discloses a real-time rendering method for accessing an AI super-division algorithm in an Android vehicle-mounted display system. The real-time rendering method comprises the following steps of judging whether a texture ID of a current frame is the same as that of a previous frame or not before each time of rendering; if the texture IDs are the same, inquiring whether a cache super-frame exists in a super-frame module or not: if the cache super-frame exists, constructing a SkiaImage and sending the SkiaImage into a rendering pipeline for display; if the cache super-framing cannot be queried, the original SkiaImage is directly used for rendering, and continuous playing is ensured; if the texture IDs are different, sending the texture IDs together with input and output hardware sharing buffer areas, the width and height of a layer to be rendered and row span information into a super-resolution implementation module for processing; after the processing of the super-division module, further querying whether the super-division module returns a super-division frame, and if the super-division frame is returned, constructing SkiaImage and sending the SkiaImage to a rendering pipeline for display; and if the super-framing is not returned, the original SkiaImage is directly used for rendering.

Description

Technical Field

[0001] This invention belongs to the field of image and video rendering technology, specifically relating to a real-time rendering method that integrates AI super-resolution algorithms into an Android in-vehicle display system. Background Technology

[0002] With the increasing demands for visual experience in in-vehicle systems, AI super-resolution algorithms are widely used for video playback, camera display, and navigation image enhancement. However, AI super-resolution models typically involve large computational loads, especially on embedded in-vehicle SoC platforms, making it impossible to process each frame of image in real time. The following problems are likely to occur during the processing: (1) Performance bottleneck: AI super-resolution latency is greater than the video frame period (1 / frame rate), resulting in some frames not being output in time. (2) Residual frame problem: During video switching or Seek operations, the super-resolution cached frames of the previous video may be displayed incorrectly. (3) Screen flicker problem: When transitioning from the original blurry frame to a clear super-resolution frame, users can perceive obvious jumps or flashbacks. (4) Blurry screen during pause: If the super-resolution frame is not output during pause, the original blurry frame is reused for rendering, leading to a decrease in experience. (5) Cross-process performance bottleneck: The AI ​​super-resolution module runs in the vendor process, while the display pipeline runs in the system process. If memory copying is used for cross-process data transfer, the performance will be further compromised. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention provides a real-time rendering method that integrates AI super-resolution algorithm into Android in-vehicle display system to solve problems such as screen flickering, blurry-to-sharp transitions, and residual frames when switching videos when performance is insufficient.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A real-time rendering method integrating AI super-resolution algorithms in an Android in-vehicle display system is characterized by its application to the SurfaceFlinger display compositor based on the Android system display framework. SurfaceFlinger is responsible for the composition and display of all layers in the system. The underlying rendering pipeline is implemented through the RenderEngine rendering engine, and the core rendering process function is RenderEngine::drawLayers(). The drawLayers() function is used to render the texture content of each layer to the Framebuffer target frame buffer using OpenGL or Skia GPU to complete the final screen output. The AI ​​super-resolution algorithm is integrated into the drawLayers() function, including the following steps: Before each rendering, check whether the texture ID of the current frame is the same as that of the previous frame; If the texture IDs are the same, check if there is a cached super-resolution frame in the super-resolution module: if a cached super-resolution frame is found, construct a SkiaImage and send it to the rendering pipeline for display; if no cached super-resolution frame is found, directly use the original SkiaImage for rendering to ensure continuous playback. If the texture IDs are different, the texture IDs, along with the input / output hardware shared buffers, the width and height of the layer to be rendered, and the row span information, are sent to the super-resolution implementation module for processing. After processing, the super-resolution module further queries whether the super-resolution module returns a super-resolution frame. If a super-resolution frame is returned, a SkiaImage is constructed and sent to the rendering pipeline for display. If no super-resolution frame is returned, the original SkiaImage is used directly for rendering.

[0005] In one possible implementation, the data sent to the super-resolution module for processing specifically includes: first, determining whether no data has been sent to the super-resolution module for N milliseconds; if so, clearing the cache; then, further determining whether the cache queue is empty; if the cache queue is empty, using an interpolation algorithm to perform real-time super-resolution amplification on the first frame of the rendering queue, and uploading the super-resolution result to the output hardware shared buffer; if the cache queue is not empty, performing AI super-resolution processing, and uploading the super-resolution result to the output hardware shared buffer.

[0006] In one possible implementation, when a video file switching or pause operation is detected, the system initiates a super-resolution module cache cleanup mechanism, including: N being an integer between 500 and 1000.

[0007] In one possible implementation, the interpolation algorithm includes bilinear interpolation.

[0008] In one possible implementation, the interpolation algorithm includes a bicubic interpolation algorithm.

[0009] The present invention has the following beneficial effects: (1) Super-resolution performance considerations: When there are repeated frames, use cached super-resolution frames for rendering to reduce CPU / NPU consumption; (2) No residual frames: The over-resolution cache is automatically cleared when switching videos or pausing; (3) Smooth transition experience: Eliminate abrupt changes and flashbacks "from blurry to clear" by using traditional interpolation fallback scheme; (4) High-performance rendering: Zero-copy communication significantly reduces CPU / GPU transmission latency and improves rendering frame rate; (5) Strong system compatibility: It can be seamlessly integrated into the Android system RenderEngine rendering pipeline and work in conjunction with Skia GPU rendering. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the steps of a real-time rendering method for integrating an AI super-resolution algorithm into an Android in-vehicle display system, according to an embodiment of the present invention. Figure 2 This is a flowchart of the super-resolution implementation module in a specific application example. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] This invention discloses a real-time rendering method for integrating an AI super-resolution algorithm into an Android in-vehicle display system. The method is applied to the SurfaceFlinger display compositor based on the Android system display framework. SurfaceFlinger is responsible for the composition and display of all layers in the system. The underlying rendering pipeline is implemented through the RenderEngine rendering engine, and the core rendering process function is RenderEngine::drawLayers(). The drawLayers() function renders the texture content of each layer into the Framebuffer target frame buffer using OpenGL or Skia GPU to complete the final screen output. An AI super-resolution algorithm is integrated into the drawLayers() function. See [link to relevant documentation]. Figure 1 This includes the following steps: Before each rendering, check whether the texture ID of the current frame is the same as that of the previous frame; If the texture IDs are the same, check if there is a cached super-resolution frame in the super-resolution module: if a cached super-resolution frame is found, construct a SkiaImage and send it to the rendering pipeline for display; if no cached super-resolution frame is found, directly use the original SkiaImage for rendering to ensure continuous playback. If the texture IDs are different, the texture IDs, along with the input / output hardware shared buffers (Ahardwarebuffer), the width and height of the layer to be rendered, and the stride information, are sent to the super-resolution implementation module for processing. After processing, the super-resolution module further checks whether the super-resolution module returns a super-resolution frame. If a super-resolution frame is returned, a SkiaImage is constructed and sent to the rendering pipeline for display. If no super-resolution frame is returned, the original SkiaImage is used directly for rendering.

[0013] Further, see Figure 2The data sent to the super-resolution module for processing includes: first, determining if no data has been sent to the super-resolution module for N milliseconds (N is an integer between 500 and 1000); if so, clearing the cache; then, checking if the cache queue is empty; if empty, using an interpolation algorithm to perform real-time super-resolution upscaling on the first frame of the rendering queue, and uploading the super-resolution result to the output hardware shared buffer (AhardwareBuffer); if the cache queue is not empty, performing AI super-resolution processing, and uploading the super-resolution result to the output hardware shared buffer (AhardwareBuffer). The interpolation algorithm can be a real-time algorithm such as bilinear or bicubic interpolation. In other words, before the first super-resolution frame returns, the first interpolated super-resolution frame is displayed, avoiding the abrupt transition from blurry to sharp and achieving a smooth visual transition.

[0014] To improve rendering performance and reduce latency, the AI ​​super-resolution module and the RenderEngine employ a zero-copy communication mechanism: shared GPU texture handles, such as hardware shared buffers (Ahardwarebuffer) / EGL images (EGLImage), are directly transferred between the system and vendor processes; the interface includes information such as input hardware shared buffers / output hardware shared buffers / stride information corresponding to input hardware shared buffers / stride information corresponding to output hardware shared buffers / input texture IDs / output texture IDs, etc.; this avoids CPU memory copying and improves the overall system frame rate and response speed.

[0015] The above-described real-time rendering method for integrating AI super-resolution algorithms into the Android in-vehicle display system, by introducing a texture ID-based frame recognition mechanism, a cache cleanup mechanism, a traditional interpolation transition strategy, and zero-copy cross-process communication, can be seamlessly integrated into the Android system's RenderEngine rendering pipeline, working collaboratively with Skia GPU rendering. Under performance constraints, it can achieve real-time clear display of video / images in the in-vehicle system, avoiding blurry abrupt changes and cross-video residual frame issues, significantly improving user experience and system smoothness.

[0016] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A real-time rendering method for integrating AI super-resolution algorithms into an Android in-vehicle display system, characterized in that, This is applied to the SurfaceFlinger display compositor based on the Android system display framework. SurfaceFlinger is responsible for the composition and display of all layers in the system. The underlying rendering pipeline is implemented through the RenderEngine rendering engine, and the core rendering process function is RenderEngine::drawLayers(). The drawLayers() function is used to render the texture content of each layer to the target frame buffer of the Framebuffer through OpenGL or Skia GPU to complete the final screen output. The drawLayers() function also incorporates an AI super-resolution algorithm, including the following steps: Before each rendering, check whether the texture ID of the current frame is the same as that of the previous frame; If the texture IDs are the same, check if there is a cached super-resolution frame in the super-resolution module: if a cached super-resolution frame is found, construct a SkiaImage and send it to the rendering pipeline for display; if no cached super-resolution frame is found, directly use the original SkiaImage for rendering to ensure continuous playback. If the texture IDs are different, the texture IDs, along with the input / output hardware shared buffers, the width and height of the layer to be rendered, and the row span information, are sent to the super-resolution implementation module for processing. After processing, the super-resolution module further queries whether the super-resolution module returns a super-resolution frame. If a super-resolution frame is returned, a SkiaImage is constructed and sent to the rendering pipeline for display. If no super-resolution frame is returned, the original SkiaImage is used directly for rendering.

2. The real-time rendering method for integrating AI super-resolution algorithm into an Android in-vehicle display system as described in claim 1, characterized in that, The specific processing steps of the super-resolution module include: first, determining whether no data has been sent to the super-resolution module for N milliseconds; if so, clearing the cache; then, determining whether the cache queue is empty; if the cache queue is empty, using an interpolation algorithm to perform real-time super-resolution amplification on the first frame of the rendering queue, and uploading the super-resolution result to the output hardware shared buffer; if the cache queue is not empty, performing AI super-resolution processing, and uploading the super-resolution result to the output hardware shared buffer.

3. The real-time rendering method for integrating AI super-resolution algorithm into an Android in-vehicle display system as described in claim 2, characterized in that, When a video file switching or pause operation is detected, the system activates the super-resolution module cache cleanup mechanism, including: N is an integer between 500 and 1000.

4. The real-time rendering method for integrating AI super-resolution algorithm into an Android in-vehicle display system as described in claim 2, characterized in that, The interpolation algorithm includes bilinear interpolation.

5. The real-time rendering method for integrating AI super-resolution algorithm into an Android in-vehicle display system as described in claim 2, characterized in that, The interpolation algorithm includes a bicubic interpolation algorithm.

Images