A container application on-screen method based on texture whole-process optimization

Abstract

The application discloses a container application on-screen method based on a texture whole-process optimization, takes a Hongmeng system as a host and an Android system as a container, the host allocates a plurality of first buffers, the container screens out a first texture format, writes original texture of the format generated by rendering into a frame buffer, identifies a dirty area of the texture after rendering, packs texture data of the area and stores the texture data into shared memory, delivers a file descriptor FD to the host, simultaneously binds the idle first buffer to set a priority, a state, a generation timestamp and a texture data pointer, and the host acquires the texture data through the FD; the host establishes a fixed thread task pool, executes a plurality of tasks in parallel, updates the state and the texture data pointer of the first buffer after obtaining on-screen texture data, finally, the host queries the first buffer with the state of being processed, reads the texture data after sorting according to the priority and the generation timestamp, completes the on-screen, and resets the state of the first buffer to be idle, thereby effectively reducing on-screen delay and improving actual frame rate of the Android application.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer software development, and particularly relates to a container application on-screen method based on texture full-process optimization. BACKGROUND

[0002] In the scenario of running an Android application through a container in a Hongmeng system, the on-screen process needs to go through four core links of generating a texture on the Android side, transmitting the texture data to the Hongmeng side, processing the texture on the Hongmeng side, and finally displaying on the screen. The current process has the following reproducible specific problems:

[0003] In terms of format conversion, the RGBA4444 format texture generated on the Android side needs to be converted into the RGBA8888 format supported by the Hongmeng system through CPU pixel by pixel. At a resolution of 1080P, the single conversion time based on a Cortex-A78 architecture is about 8ms, accounting for 48% of a single frame period; in terms of data transmission, even if only 10% of the area changes between frames, the complete texture data still needs to be transmitted. At a resolution of 2K, the single transmission amount of the full-amount RGBA8888 texture is 14.4MB, and the occupied memory bandwidth is about 864MB / s at 60fps, resulting in a transmission time of no less than 16ms, causing a waste of transmission bandwidth; in terms of serial processing, the Android side rendering needs 10ms, the transmission process needs 16ms, the Hongmeng side processing needs 8ms, and finally the on-screen needs 2ms, so the total serial time is 36ms, which is far more than the frame period of 16.7ms, which directly leads to a frame rate of 27fps, forming a serial process blockage; in terms of static texture processing, UI icons are repeatedly transmitted and drawn every frame, with a single frame added transmission amount of 0.16MB, a cumulative bandwidth of 9.6MB / s, and a drawing call time of 0.5ms each time. The repeated processing of such static textures will cause necessary processing time.

[0004] In summary, the existing on-screen process still has many problems in format conversion, data transmission, serial processing and static texture processing, resulting in a large delay in the on-screen process. SUMMARY

[0005] Therefore, the application provides a container application on-screen method based on texture full-process optimization, which realizes low-delay on-screen of the container application through format negotiation, incremental transmission, parallel processing and asynchronous on-screen operation.

[0006] The container application on-screen method based on texture full-process optimization provided by the application specifically includes the following steps:

[0007] The Hongmeng system is used as a host and the Android system is used as a container.

[0008] When the system starts, the host allocates a plurality of first buffers containing a buffer ID, a texture data pointer, a state, a priority and a generation timestamp; the container filters out a first texture format in a manner that a compressed format is superior to a non-compressed format and a lower decompression delay is more preferred, from the intersection of the texture formats supported by the host and the texture formats supported by the container itself, and writes the original texture of the first texture format into a frame buffer after rendering is completed;

[0009] The container obtains a dirty region mask in the frame buffer to generate a dirty region, packs the texture data of the dirty region and saves it to shared memory, and passes the FD of the shared memory to the host; a first buffer with an idle state is obtained, the priority is set, the state is set to rendered, the generation timestamp is recorded, and the address index of the shared memory texture data is set as the texture data pointer;

[0010] The host obtains the texture data through the shared memory FD, creates a task pool of fixed threads, establishes a task structure containing a task type, a texture data pointer and a task completion semaphore; the texture data pointer of the first buffer with a rendered state is obtained, and the task is executed in parallel to obtain up-screenable texture data, the state of the first buffer is set to processed, and the texture data pointer is updated to the address of the up-screenable texture data;

[0011] The host obtains the first buffer with a processed state, sorts according to the priority of the first buffer, sorts according to the generation timestamp if the priority of the first buffer is the same, reads the texture data pointer in turn according to the sorting to obtain up-screenable texture data and performs up-screening operation, and resets the state of the first buffer to idle after the execution is completed.

[0012] Further, the texture data of the dirty region is packed in a structured format according to the number of regions, region coordinates and region pixel data.

[0013] Further, the priority is set in a manner that the priority of the first buffer is set according to the scene related to the texture data, the highest priority is set for the scene related to high dynamic, the medium priority is set for the scene related to video, and the lowest priority is set for the scene related to static UI.

[0014] Further, the first texture format is filtered out in a manner that the host sends a format list formed by the texture formats supported by the host to the container system through an IPC interface, the format list includes parameters of the texture format; the container obtains the texture formats and texture compression formats supported by the container, sorts the texture formats supported by the container and the format list in a manner that the decompression delay is from low to high and all texture compression formats are arranged before non-compressed texture formats, and the format arranged in the first place is taken as the first texture format.

[0015] Further, the selected first texture format is dynamically adjusted at runtime in the following manner: the container periodically acquires the current rendering frame rate, and if the frame rate is not greater than a threshold value, the first texture format is the format with the highest compression ratio, otherwise the first texture format is the format with the lowest decompression delay; when the GPU load of the host is greater than a threshold value, the container sends a message to the host to switch the first texture format to a non-compressed format.

[0016] Further, when the intersection of the texture formats supported by the host and the container is empty, the container filters the first texture format according to the decompression delay from low to high and all texture compression formats are arranged before the non-compressed texture format, and in rendering, the first texture format is used as the color attachment binding conversion frame buffer, the original texture of the first texture format is converted and expanded by the conversion shader, the original texture is bound to draw a full-screen graph, and the GPU converts it to the format supported by the host and writes it to the frame buffer.

[0017] Further, the conversion and expansion of the original texture of the first texture format by the conversion shader is as follows: the vertex shader converts the vertex coordinates of the full-screen quadrilateral of the original texture into normalized coordinates required for texture sampling, and the fragment shader performs texture sampling on each pixel, restores the precision of the first texture format, and expands it to the precision of the format supported by the host.

[0018] Further, it further comprises identifying and caching static textures, specifically:

[0019] The container records the texture ID and drawing area of each frame of data, and when a continuous set of a certain number of frames of data meet the conditions that the texture ID is unchanged and the change in the drawing area is less than a threshold value, these frames of data are marked as static textures; the hash value of the pixel data, width, height and format of the static texture is calculated as the unique identifier of the static texture;

[0020] The host constructs a video memory cache pool and a memory cache pool, wherein the video memory cache pool stores high-frequency access static textures, and the core fields include a unique identifier, a corresponding texture ID of the host, a width and a height, an access frequency and a last access time, and the memory cache pool stores low-frequency access static textures in a compressed format;

[0021] When the host receives new texture data or performs an on-screen operation, the hash value of the received texture is calculated, and the video memory cache pool and the memory cache pool are traversed, if the hash value matches, for high-frequency textures, the texture ID of the video memory cache is reused, and for low-frequency textures, the texture is decompressed from the memory cache and reused; the on-screen thread binds the texture ID of the cache and directly performs the on-screen operation.

[0022] Further, the host uses a rectangular packing algorithm to sort the static textures in descending order of area, packs the static textures with sizes less than the threshold into a atlas canvas, records the coordinates of the sub-textures in the atlas canvas and generates a mapping table.

[0023] Further, the task pool of the fixed thread comprises three types of tasks, specifically: a data checking task, used for checking whether the number of regions and the data size are matched, checking the data integrity of the compressed texture, and returning an error code if the checking fails; a compressed texture decompression task, used for decompressing the compressed format texture by calling a hardware decompression interface, inputting the format, data source, target buffer, width and height to complete the decompression; and an incremental region merging task, used for binding the base texture canvas and the incremental texture, enabling the GPU copy function, copying each incremental region to the corresponding position of the base canvas, and updating the multi-level detail map of the texture. Beneficial effects

[0024] The application takes the Hongmeng system as a host and the Android system as a container, the host system allocates a plurality of system memory level first buffers containing buffer ID, texture data pointer and the like to construct a buffer queue, the container system screens a first texture format and writes the original texture of the format generated by rendering into a frame buffer; the container system identifies the dirty region of the texture after rendering, packs the texture data of the region into a shared memory, and delivers the file descriptor FD thereof to the host system, while binding the idle first buffer to set the priority, state, generation timestamp and texture data pointer, the host system creates a DMA transmission channel to obtain the texture data through the FD; the host system establishes a fixed thread task pool bound to a high-performance core, and executes the data checking, compressed texture decompression and incremental region merging tasks in parallel, updates the state and texture data pointer of the first buffer after obtaining the up-screenable texture data by processing, and finally the host system polls the first buffer with the processed state, reads the texture data to complete the up-screening after sorting according to the priority and generation timestamp, and resets the state of the first buffer to idle, thereby effectively reducing the up-screening delay and improving the actual frame rate of the Android application. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A flowchart of a container application up-screening method based on texture full-process optimization provided by the application is shown. DETAILED DESCRIPTION

[0026] The application will be described in detail below with reference to the embodiments and the accompanying drawings.

[0027] The application provides a container application on-screen method based on a texture full-process optimization, and the core idea is that a Hongmeng system is used as a host, an Android system is used as a container, the host system allocates a plurality of system memory level first buffers containing buffer ID, texture data pointer and other information to build a buffer queue, the container system screens a first texture format to complete format negotiation, and writes the first texture format and original texture of the format generated by rendering into a frame buffer; the container system identifies a dirty area of the texture after rendering, packs the area texture data into a shared memory, and delivers a file descriptor FD thereof to the host system, while binding the idle first buffer to set a priority, a state, a generation timestamp and a texture data pointer, the host system creates a DMA transmission channel to obtain the texture data through the FD; the host system establishes a fixed thread task pool bound to a high-performance core, and executes data verification, texture decompression, and incremental area merging tasks in parallel, updates the state and the texture data pointer of the first buffer after processing the obtained on-screen texture data; and finally, the host system polls the first buffer with the state of being processed, reads the texture data to complete on-screen display after sorting according to the priority and the generation timestamp, and resets the state of the first buffer to idle.

[0028] The application provides a container application on-screen method based on a texture full-process optimization, and the core idea is that a Hongmeng system is used as a host, an Android system is used as a container, the host system allocates a plurality of system memory level first buffers containing buffer ID, texture data pointer and other information to build a buffer queue, the container system screens a first texture format to complete format negotiation, and writes the first texture format and original texture of the format generated by rendering into a frame buffer; the container system identifies a dirty area of the texture after rendering, packs the area texture data into a shared memory, and delivers a file descriptor FD thereof to the host system, while binding the idle first buffer to set a priority, a state, a generation timestamp and a texture data pointer, the host system creates a DMA transmission channel to obtain the texture data through the FD; the host system establishes a fixed thread task pool bound to a high-performance core, and executes data verification, texture decompression, and incremental area merging tasks in parallel, updates the state and the texture data pointer of the first buffer after processing the obtained on-screen texture data; and finally, the host system polls the first buffer with the state of being processed, reads the texture data to complete on-screen display after sorting according to the priority and the generation timestamp, and resets the state of the first buffer to idle. Figure 1

[0029] Step 1, a Hongmeng system is used as a host system, and an Android system is used as a container system; when the system starts, the host system allocates a plurality of system memory level first buffers to build a buffer queue, the first buffer contains buffer ID, texture data pointer, state, priority, generation timestamp and mutual exclusion lock and other information, wherein the state includes idle, rendered and processed; the container system screens a first texture format in the intersection of the texture format supported by the host system and the texture format supported by the container system, and completes format negotiation by selecting the first texture format in the manner that the compressed format is superior to the non-compressed format and the lower the decompression delay is, the higher the priority is, and sends the first texture format to the host system; after the container system completes rendering, the container system writes the original texture of the first texture format generated by rendering into a frame buffer.

[0030] The manner of screening the first texture format is that the host system sends a format list formed by the texture format supported by the host system to the container system through an IPC interface, the format list includes the parameters of the texture format; the container system obtains the supported texture format and the texture compression format, and sorts the supported texture format and the format list in the manner that the decompression delay is from low to high and all the texture compression formats are arranged before the non-compressed texture format, and takes the format arranged in the first place as the first texture format.

[0031] Further, the application dynamically adjusts the selected first texture format at runtime, and the specific manner is that:

[0032] ​The container system periodically acquires a current rendering frame rate, and if the frame rate is not greater than a threshold value, adopts a format with the highest compression ratio as the first texture format, otherwise adopts a format with the lowest decompression delay as the first texture format.

[0033] When the GPU load of the host system is greater than a threshold value, a message is sent to the container system to switch the first texture format to a non-compressed format.

[0034] Further, when the intersection is empty, the container system filters out the first texture format in a manner that the decompression delay is from low to high and all texture compression formats are arranged before the non-compressed texture format, and in rendering, the first texture format is used as a color attachment binding conversion frame buffer, the original texture of the first texture format is precision-converted and expanded by a conversion shader, the original texture is bound to draw a full-screen graphics, and the GPU converts it into a format supported by the host system and writes it into the frame buffer.

[0035] Further, the implementation manner of the container system in which the conversion shader performs precision conversion and expansion on the original texture of the first texture format is that a vertex shader converts the vertex coordinates of a full-screen quadrilateral of the original texture into normalized coordinates required for texture sampling, providing a position reference for texture sampling of each pixel in the subsequent; a fragment shader performs texture sampling on each pixel, restores the precision of the first texture format, and expands it into the precision of a format supported by the host system.

[0036] Step 2, the container system enables the extension function of the GPU frame buffer, obtains the dirty region mask in the frame buffer after each frame of data is rendered, parses and generates a dirty region list represented by screen coordinates; reads the texture data of the dirty region included in the dirty region list in the frame buffer, packs the texture data according to the structured format of the region number, region coordinates and region pixel data, saves the packed texture data to the shared memory, and passes the file descriptor FD of the shared memory to the host system; obtains a first buffer with an idle state, sets the priority of the first buffer according to the scene related to the texture data, sets the state of the first buffer to rendered, records the generation timestamp, sets the address index of the shared memory texture data as the texture data pointer; the host system creates a memory-to-memory DMA transmission channel and obtains the texture data through the shared memory FD.

[0037] The priority of the first buffer is set according to the scene related to the texture data, specifically: the high dynamic scene is set to the highest priority, the video scene is set to the medium priority, and the static UI scene is set to the lowest priority.

[0038] Step 3, the host system creates a task pool of fixed threads matched with the number of CPU cores, binds the threads to high-performance cores, defines three task types of data verification, compressed texture decompression and incremental region merging, establishes a task structure containing a task type, a texture data pointer and a task completion semaphore, polls a buffer queue to obtain a texture data pointer of a first buffer record in a rendered state, constructs tasks of the three task types, executes the three types of tasks in parallel, completes data verification, compressed texture decompression and incremental region merging processing on the obtained texture data, obtains complete and on-screen texture data, sets the state of the first buffer to processed, and updates the texture data pointer to the address of the on-screen texture data.

[0039] The data verification task is used for checking whether the region quantity and the data size are matched, checking the data integrity of the compressed texture, and returning an error code if the checking fails; the compressed texture decompression task is used for calling a hardware decompression interface, inputting parameters such as a format, a data source, a target buffer, a width and the like, and completing decompression on the compressed format texture; and the incremental region merging task is used for binding a base texture canvas and an incremental texture, enabling a GPU copy function, copying each incremental region to a corresponding position of the base canvas, and updating a texture multi-level detail map.

[0040] Step 4, the host system polls a buffer queue to obtain a first buffer in a processed state, sorts the first buffer according to a priority of the first buffer, sorts the first buffers with the same priority according to a generation timestamp, sequentially obtains on-screen texture data according to a texture data pointer recorded in the first buffer, and executes an on-screen operation, and resets the state of the first buffer to idle after the execution is completed.

[0041] In order to further improve the transmission and drawing efficiency of the texture, the application identifies and caches the static texture, specifically including:

[0042] The container system records a texture ID and a drawing region of each frame of data, monitors content update of each frame of data, marks the frame of data as a static texture when a continuous set number of frames of data meet the conditions that the texture ID is unchanged and a change amount of the drawing region is less than a threshold value, calculates a hash value of pixel data, width, height and format of the static texture, and the hash value is a unique identifier of the static texture.

[0043] A video memory cache pool and a memory cache pool are constructed in the host system, the video memory cache pool stores high-frequency accessed static textures, core fields include a hash value of the static texture, a texture ID of the corresponding host system, width and height, access times and last access time, and the video memory cache pool can be directly used for on-screen drawing, and the memory cache pool stores low-frequency accessed static textures, stores data in a compressed format, and needs to be decompressed and reused.

[0044] When the host system receives new texture data or performs the on-screen operation, the hash value of the received texture is calculated, the video memory cache pool and the memory cache pool are traversed, if the hash values match, they are directly reused, the high-frequency texture reuses the texture ID of the video memory cache, and the low-frequency texture is decompressed from the memory cache and reused; the on-screen thread binds the static texture ID of the cache, directly performs the on-screen operation, and avoids the overhead of repeated drawing.

[0045] Further, for static textures with a size less than a threshold, the host system sorts the static textures in descending order of area according to a rectangular packing algorithm, packs them into an atlas canvas, records the coordinates of the sub-textures in the atlas canvas, and generates a mapping table; the on-screen thread binds the atlas canvas texture, and displays multiple sub-textures through one drawing call, thereby reducing the number of GPU drawing calls. Embodiments

[0046] This embodiment takes OpenHarmony as the host system as an example, adopts a container application on-screen method based on texture full-process optimization provided by the application, solves the problem of low frame rate of Android application caused by high on-screen delay through the design of texture generation, transmission, processing, scheduling and reuse links, and is suitable for mobile terminal cross-system container graphics performance optimization scenarios, and the specific process includes:

[0047] S1, in the texture generation stage, complete cross-system texture format dynamic negotiation and hardware acceleration conversion, generate OpenHarmony side compatible texture format when rendering on the Android side, avoid CPU conversion, complete necessary format conversion through hardware acceleration, specifically including:

[0048] S1.1, format negotiation in the host system startup stage.

[0049] S1.1.1, the data structure is defined as follows:

[0050] / / OpenHarmony side supported format information structure (transmitted through IPC)

[0051] typedef struct {

[0052] int format; / / Format enumeration (such as OH_FORMAT_RGBA8888=1, OH_FORMAT_ASTC_4x4=2)

[0053] bool isCompressed; / / Whether it is a compressed format

[0054] int maxWidth; / / Maximum width supported by the format

[0055] int maxHeight; / / The maximum height supported by this format

[0056] int decompressLatency; / / Hardware decompression latency (in μs, used for priority sorting)

[0057] } OH_SupportedFormat;

[0058] / / Android side response negotiation result structure

[0059] typedef struct {

[0060] int selectedFormat; / / Selected format

[0061] bool needConversion; / / Whether format conversion is needed

[0062] } Android_FormatNegotiationResult;

[0063] S1.1.2 The format negotiation process includes:

[0064] On the OpenHarmony side, a list of supported formats is sent via the container IPC interface OH_Container_SendSupportedFormats, including format parameters such as decompressLatency=500μs for OH_FORMAT_ASTC_4x4. The format list can contain a maximum of eight formats. On the Android side, glGetIntegerv(GL_NUM_COMPRESSED_TEXTURE_FORMATS, &count) is called to obtain its supported compression formats, and glGetIntegerv(GL_IMPLEMENTATION_COLOR_READ_FORMAT, &nativeFormat) is called to obtain native uncompressed formats. On the Android side, compatible formats are sorted according to priority rules, compressed formats are sorted in ascending order of decompressLatency, and uncompressed formats are sorted according to the native formats of both parties, with all compressed formats preceding uncompressed formats. The format ranked first is selected. The Android side returns the selected format, such as selectedFormat=OH_FORMAT_ASTC_4x4, via Android_Container_SendNegotiationResult.

[0065] S1.2, runtime dynamic adjustment, Android side gets the current rendering frame rate every 100 ms through Choreographer.getInstance().getFrameRate(), if the frame rate is not greater than 30 fps, it is a static scene, and then switch to a format with higher compression ratio, such as ASTC_6x6 with a compression ratio of 8:1; if the frame rate is greater than 30 fps, it is a dynamic scene, and then switch to a format with faster decompression, such as ASTC_4x4 with a decompression delay of 500 μs; OpenHarmony side gets GPU load through OH_GPU_GetLoad(), when GPU load is greater than 80%, actively sends OH_Container_RequestFormatChange through IPC to require Android side to switch to non-compressed format to reduce the pressure of GPU decompression.

[0066] S1.3, GPU hardware acceleration format conversion, when it must be converted, such as Android side only supports RGBA4444 and OpenHarmony side format is RGBA8888, then execute: Android side creates a conversion FBO, such as glGenFramebuffers(1, &convertFbo), binds a 2D texture as a color attachment with format RGBA8888; compile and link the conversion shader, the shader code is as follows:

[0067] / / Vertex shader passes texture coordinates

[0068] attribute vec2 a_position;

[0069] varying vec2 v_texCoord;

[0070] void main() {

[0071] gl_Position = vec4(a_position, 0.0, 1.0);

[0072] v_texCoord = (a_position + 1.0) / 2.0; / / Normalize coordinates

[0073] }

[0074] / / Fragment shader converts format RGBA4444 to format RGBA8888

[0075] precision mediump float;

[0076] varying vec2 v_texCoord;

[0077] uniform sampler2D u_srcTexture; / / Input RGBA4444 texture

[0078] void main() {

[0079] vec4 color = texture2D(u_srcTexture, v_texCoord); / / Read value range [0,1]

[0080] gl_FragColor = vec4(floor(color.r * 15.0) / 15.0, / / Restore 4-bit precision (0-15)

[0081] floor(color.g * 15.0) / 15.0,

[0082] floor(color.b * 15.0) / 15.0,

[0083] floor(color.a * 15.0) / 15.0) * 255.0; / / Expand to 8 bits

[0084] }

[0085] Bind the original RGBA4444 texture to u_srcTexture, and use glDrawArrays(GL_TRIANGLE_STRIP, 0, 4) to draw a full-screen quadrilateral. The GPU automatically completes the conversion and writes it to the FBO's RGBA8888 texture.

[0086] The converted texture is stored in ION shared memory, which is created using Android GraphicBuffer, with the format HAL_PIXEL_FORMAT_RGBA_8888 and permissions GRALLOC_USAGE_HW_TEXTURE | GRALLOC_USAGE_SHARED.

[0087] S2. In the texture transmission stage, implement hardware-assisted dirty region prediction and incremental texture transmission, transmitting only texture regions that change between frames. Combined with prediction, the amount of data transmitted is reduced, and transmission is accelerated through DMA. Specifically, this includes:

[0088] S2.1 Hardware-level dirty region tracking.

[0089] High-End Adreno GPU implementation: enable ADRENO_FRAMEBUFFER_DELTA extension glEnable(0x8A50), configure comparison mode as block-level difference glAdrenoFramebufferDeltaMode(0x8A51, 0x01), 16x16 pixel block; after each frame rendering is completed, call glAdrenoGetDeltaMask(0x8A52, &maskBuffer, &maskSize) to obtain the dirty region mask, each bit indicates whether a 16x16 block has changed; parse the mask, traverse maskBuffer, and convert the block coordinates of the bit 1 (x is block index%(width / 16), y is block index / (width / 16)) to screen coordinates (x16, y16, 16, 16) to form a dirty region list Rect[] dirtyRegions.

[0090] ARM Mali GPU implementation: enable ARM_framebuffer_compression extension, bind the previous frame FBO as the reference frame glBindFramebuffer(ARM_REFERENCE_FRAMEBUFFER, prevFbo); call glGetFramebufferParameteriv(ARM_FRAMEBUFFER_DIRTY_REGIONS, &count, regions) to directly obtain the dirty region list.

[0091] S2.2, dynamic region prediction.

[0092] The data structure is as follows:

[0093] class RegionTrack {

[0094] Rect region; / / region coordinates

[0095] long timestamp; / / appearance timestamp

[0096] int dx; / / x direction movement (pixels / frame)

[0097] int dy; / / y direction movement (pixels / frame)

[0098] }

[0099] The prediction process is: maintaining a dirty region track list List of the last 3 frames; for the same region appearing continuously, IoU >= 0.8, calculating the movement: dx = (r3.x-r1.x) / 2, dy = (r3.y-r1.y) / 2; predicting the next frame region: predictedRegion = new Rect(r3.x+dx, r3.y+dy, r3.width, r3.height), extending the 10% boundary predictedRegion.inset(-r3.width0.1, -r3.height0.1); merging the predicted region and the dirty region tracked by hardware, that is, deduplicating and merging the overlapping regions to generate the final transmission region Rect[] finalRegions.

[0100] S2.3, incremental data packaging and DMA transmission.

[0101] The data packaging format is: [number of regions (4B)] + [region 1 (x1, y1, w1, h1, each 4B)] + [region 1 pixel data] +... + [region n pixel data], the packaging method is: the Android side calls glReadPixels(x, y, w, h, GL_RGBA, GL_UNSIGNED_BYTE, pixelBuffer) for each region, binds glFenceSync to ensure data readiness: glClientWaitSync(sync, 0, 1000000); for ASTC compressed textures, the dirty region corresponding compressed block is extracted according to 4x4 block index, and glGetCompressedTexImage is used to read only block data instead of full amount. The DMA transmission is: the OpenHarmony side creates a DMA channel through OH_Dma_CreateChannel(DMA_CHANNEL_MEM_TO_MEM); the Android side passes the shared memory FD to the OpenHarmony side through ion_share_fd; OH_Dma_StartTransfer(channel, srcAddr, dstAddr, dataSize) is called to start DMA transmission, and OH_Dma_GetCompletionStatus is used to notify the completion of transmission.

[0102] S3, texture processing stage, parallel processing of texture reception and hardware decompression on the OpenHarmony side, receiving texture data through multi-thread parallel processing, and using GPU to accelerate decompression and region merging.

[0103] S3.1, multi-threaded task pool initialization, create a task pool of 3 fixed threads, bind the thread to high-performance computing core CPU2 and CPU3 through OH_Thread_SetAffinity(threadId, 0x0C), and the task type and processing function mapping is as follows:

[0104] typedef enum {

[0105] TASK_CHECK = 0, / / data check

[0106] TASK_DECOMPRESS = 1, / / compressed texture decompression

[0107] TASK_MERGE = 2 / / incremental region merge

[0108] } TaskType;

[0109] / / task structure

[0110] typedef struct {

[0111] TaskType type;

[0112] TextureData* data; / / pointer to texture data

[0113] sem_t* doneSem; / / task completion semaphore

[0114] } TextureTask;

[0115] S3.2, parallel task processing.

[0116] Data check task (TASK_CHECK), check if the number of regions matches the data size (sum(wh4)==dataSize-4-16*n); for compressed textures, check the block CRC by OH_CRC_Calculate(blockData,blockSize)==blockCrc; if the check fails, return error code -1 through sem_post(doneSem).

[0117] Compressed texture decompression task (TASK_DECOMPRESS): if it is in ASTC format, call the hardware decompression interface as follows:

[0118] OH_GPU_DecompressParam param = {

[0119] .format = OH_FORMAT_ASTC_4x4,

[0120] .src = compressedData,

[0121] .srcSize = compressedSize,

[0122] .dst = decompressedBuffer,

[0123] .width = 2560,

[0124] .height = 1440

[0125] };

[0126] OH_GPU_DecompressTexture(¶m); / / Hardware decompression, takes about 1ms (2K scene)

[0127] Incremental region merge task (TASK_MERGE): bind the base texture canvas glBindTexture(GL_TEXTURE_2D, baseTextureId) and the incremental texture glBindTexture(GL_TEXTURE_2D, deltaTextureId); enable GPU Blit with glEnable(GL_BLIT_FRAMEBUFFER), and for each region, perform the following operations:

[0128] glBlitFramebuffer(

[0129] 0, 0, region.w, region.h, / / Source region (incremental texture)

[0130] region.x, region.y, region.x+region.w, region.y+region.h, / / Target region (base canvas)

[0131] GL_COLOR_BUFFER_BIT, GL_NEAREST / / Copy color buffer );

[0133] After merging, call glGenerateMipmap(GL_TEXTURE_2D) to update the texture mipmap.

[0134] S3.3 Task Synchronization and Error Handling: The main thread waits for all tasks to complete via sem_wait, with doneSem counted as 3. If any task fails, a retry mechanism is triggered, i.e., the complete texture of that frame is re-requested, with a maximum of 2 retries.

[0135] S4. In the screen display stage, an asynchronous screen display mechanism based on multi-buffered queues and priority scheduling is used. Rendering, processing, and screen display are all performed in parallel through 3-buffered queues. The screen display timing is optimized in conjunction with the V-Sync signal. The specific process is as follows:

[0136] S4.1, Three-buffered queue design, each buffer contains the following data:

[0137] typedef struct {

[0138] int bufferId; / / 0 / 1 / 2

[0139] void* textureData; / / Texture data pointer

[0140] enum { BUFFER_IDLE, BUFFER_RENDERED, BUFFER_PROCESSED} status; / / status

[0141] int priority; / / Priority (1-3)

[0142] int64_t timestamp; / / Generate timestamp

[0143] pthread_mutex_t lock; / / Mutex lock

[0144] ScreenBuffer;

[0145] ScreenBuffer buffers[3] = {

[0146] {.bufferId=0, .status=BUFFER_IDLE, .priority=3},

[0147] {.bufferId=1, .status=BUFFER_IDLE, .priority=3},

[0148] {.bufferId=2, .status=BUFFER_IDLE, .priority=3}

[0149] };

[0150] S4.2, full parallel process, when rendering on the Android side, call findIdleBuffer() to get the buffer with status BUFFER_IDLE, set status = BUFFER_RENDERED, priority = 1 (game) or 3 (static UI), and timestamp = systemTime() after rendering is completed;

[0151] OpenHarmony side processing, polling buffers, getting buffers with status BUFFER_RENDERED, and setting status = BUFFER_PROCESSED after processing is completed;

[0152] The on-screen thread binds the V-Sync signal through OH_Display_RegisterVSyncCallback(vsyncCallback); in the callback function, select the earliest generated buffer from the buffers with status BUFFER_PROCESSED according to priority (1>2>3); call OH_Window_AttachTexture(window, buffer->textureData) to on-screen, and set status = BUFFER_IDLE after completion.

[0153] S4.3, priority scheduling and degradation strategy. Set priority: 1 for game application, frame rate greater than 30fps; 2 for video, frame rate between 24 and 30fps; 3 for static UI, frame rate less than 24fps. When the low-priority buffer wait timeout is greater than 33ms, forcibly promote the priority to 2; get the time to the next V-Sync through OH_Display_GetNextVSyncTime(), if less than 5ms and no BUFFER_PROCESSED buffer, then use the last frame buffer lastBufferId to on-screen.

[0154] S5, multiplexing stage, static texture intelligent cache pool and graph merging and reuse, used to identify and cache static textures, reduce drawing calls through graph merging, the specific process is as follows:

[0155] S5.1, static texture identification, the Android side intercepts the OpenGL ES command stream through the EGLHook hook, records the texture ID of glBindTexture and the drawing area (x, y, w, h) of glDrawArrays of each frame, and marks the texture as static if the following conditions are met for 5 consecutive frames: texture ID is unchanged, drawing area changes less than 1 pixel, and glTexSubImage2D is not called; generate a unique identifier: hash = SHA256(texture pixel data + width + height + format).

[0156] S5.2, multi-level cache pool implementation, video memory cache pool capacity is 8MB, the structure is as follows:

[0157] typedef struct {

[0158] char hash

[32] ; / / SHA256 hash

[0159] uint32_t textureId; / / OpenHarmony side texture ID

[0160] int width, height;

[0161] int accessCount; / / Access times

[0162] int64_t lastAccessTime; / / Last access time

[0163] } GpuCacheItem;

[0164] GpuCacheItem gpuCache

[100] ; / / Maximum 100 entries

[0165] Memory cache pool capacity is 32MB: store compressed texture data (ASTC format), structure is the same but replace textureId with compressedData pointer.

[0166] Query process, calculate hash when receiving texture and traverse cache pool, if matched, directly reuse textureId (video memory cache) or decompress and reuse (memory cache).

[0167] S5.3, texture atlas merging. For static small textures with size not greater than 256x256, perform rectangular packing algorithm: sort textures by area in descending order, large textures are placed first; atlas is initialized to 1024x1024, place from the top left corner, if the remaining space is insufficient, create a new atlas; record the coordinates (atlasX, atlasY, w, h) of each sub-texture in the atlas, generate a mapping table map<textureHash, AtlasCoord>; when uploading, bind the atlas texture, draw multiple sub-textures at a time by modifying the vertex coordinates to the coordinates in the atlas.

[0168] S5.4, cache eviction strategy: when the cache capacity is over limit, evict the entry meeting the following conditions: lastAccessTime is less than the difference between the current time and 30000 ms, i.e., 30 seconds have not been accessed, accessCount is less than 5, if the sub-texture eviction ratio is greater than 50%, only the remaining sub-textures are retained and the atlas is reconstructed.

[0169] It is verified by experiments that the on-screen delay of the Android application is reduced from 36 ms to not more than 10 ms, the actual frame rate of the Android application is increased from 27 fps to 58 fps, the texture transmission bandwidth is reduced from 864 MB / s to 103 MB / s, and the GPU rendering call number is reduced by 60% under a static UI scene.

[0170] To sum up, the above is only a preferred embodiment of the present application, not for limiting the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for displaying container applications based on full-process texture optimization, characterized in that, Specifically, the following steps are included: Using HarmonyOS as the host and Android as the container; When the system starts, the host allocates multiple first buffers, each containing a buffer ID, texture data pointer, status, priority, and generation timestamp. The container selects the first texture format from the intersection of the texture formats supported by the host and the texture formats it supports, prioritizing compressed formats over uncompressed formats and lower decompression latency. After rendering, the original texture of the first texture format is written to the frame buffer. The container obtains the dirty region mask in the framebuffer to generate the dirty region, packs the texture data of the dirty region and saves it to shared memory, and passes the file descriptor FD of the shared memory to the host. Get the first buffer that is idle, set its priority, set its status to rendered, record the generation timestamp, and set the address index of the shared memory texture data to the texture data pointer. The host obtains texture data through the file descriptor FD in shared memory, creates a task pool with fixed threads, and establishes a task structure containing task type, texture data pointer, and task completion semaphore; obtains the texture data pointer of the first buffer whose state is rendered, executes tasks in parallel to obtain texture data that can be displayed on the screen, sets the state of the first buffer to processed, and updates the texture data pointer to the address of the texture data that can be displayed on the screen. The host obtains the first buffer with the status of being processed, sorts them according to their priority, and if the priorities are the same, sorts them according to their generation timestamp. It reads the texture data pointers in order according to the sorting to obtain the texture data that can be displayed on the screen and performs the display operation. After the operation is completed, the status of the first buffer is reset to idle.

2. The container application display method according to claim 1, characterized in that, The method for packaging the texture data of the dirty region is as follows: the texture data is packaged according to a structured format of region number, region coordinates and region pixel data.

3. The container application display method according to claim 1, characterized in that, The priority setting method is as follows: the priority of the first buffer is set according to the scene related to the texture data, the high dynamic scene related is set to the highest priority, the video scene related is set to the medium priority, and the static UI scene related is set to the lowest priority.

4. The container application display method according to claim 1, characterized in that, The method for selecting the first texture format according to the principle of prioritizing compressed formats over uncompressed formats and lower decompression latency is as follows: The host sends a list of supported texture formats to the container system via the IPC interface. The list of formats includes the parameters of the texture formats. The container obtains the texture formats and texture compression formats it supports, sorts the supported texture formats and the list of formats according to the principle of decompression latency from low to high and all compressed texture formats are ranked before uncompressed texture formats, and selects the format ranked first as the first texture format.

5. The container application display method according to claim 1, characterized in that, The selected first texture format is dynamically adjusted at runtime. Specifically, the container periodically obtains the current rendering frame rate. If the frame rate is not greater than the threshold, the format with the highest compression ratio is used as the first texture format. Otherwise, the format with the lowest decompression latency is used as the first texture format. When the host's GPU load is greater than the threshold, a message is sent to the container to switch the first texture format to an uncompressed format.

6. The container application display method according to claim 1, characterized in that, When the intersection of the texture formats supported by the host and the texture formats supported by the container is empty, the container selects the first texture format in order of decompression latency from low to high and all compressed texture formats are ranked before uncompressed texture formats. During rendering, the first texture format is used as the color attachment to bind the frame buffer for conversion. The conversion shader performs precision conversion and expansion on the original texture of the first texture format, binds the original texture to draw full-screen graphics, and then the GPU converts it to a format supported by the host and writes it to the frame buffer.

7. The container application display method according to claim 6, characterized in that, The method by which the conversion shader performs precision conversion and expansion on the original texture of the first texture format is as follows: the vertex shader converts the vertex coordinates of the full-screen quadrilateral of the original texture into normalized coordinates required for texture sampling, the fragment shader performs texture sampling on each pixel, restores the precision of the first texture format, and then expands it to the precision of the format supported by the host.

8. The container application display method according to claim 1, characterized in that, This also includes the identification and caching of static textures, specifically: The container records the texture ID and drawing area of ​​each frame of data. When a set number of consecutive frame data satisfy the condition that the texture ID remains unchanged and the change in the drawing area is less than the threshold, these frame data are marked as static textures. Calculate the pixel data, width, height, and format hash value of the static texture as a unique identifier for the static texture; The host constructs a video memory cache pool and a memory cache pool. The video memory cache pool stores frequently accessed static textures. The core fields include a unique identifier, the corresponding host texture ID, width and height, access count, and last access time. The memory cache pool stores infrequently accessed static textures in a compressed format. When the host receives new texture data or performs an on-screen operation, it calculates the hash value of the received texture, traverses the video memory cache pool and the memory cache pool, and if the hash values ​​match, it reuses the texture ID of the video memory cache for high-frequency textures and decompresses and reuses it from the memory cache for low-frequency textures. The thread that displays the image binds to the cached texture ID and directly performs the display operation.

9. The container application display method according to claim 8, characterized in that, For static textures with a size smaller than the threshold, the host uses a rectangular binning algorithm to sort the static textures in descending order of area, packs them into the atlas canvas, records the coordinates of the sub-textures in the atlas canvas, and generates a mapping table. The on-screen thread is bound to the atlas canvas texture, and multiple sub-textures are displayed by a single drawing call.

10. The container application on-screen method according to claim 1, characterized in that, The task pool for creating fixed threads contains three types of tasks: data verification tasks, which are used to check whether the number of regions matches the data size, verify the integrity of compressed textures, and return an error code if the verification fails; and compressed texture decompression tasks, which are used to decompress compressed textures by calling the hardware decompression interface and passing in the format, data source, target buffer, width and height. The incremental region merging task is used to bind the base texture canvas and the incremental texture, enable the GPU copy function, copy each incremental region to the corresponding position on the base canvas, and update the multi-level detail map of the texture.

Images