Method and device for asynchronously updating multiple channels of image data

Abstract

The present application relates to the field of data processing, and provides a method and device for asynchronously updating multiple channels of image data. The method includes: creating multiple graphics buffer memory blocks and multiple external extension textures; based on the application extension interface, establishing a binding relationship between each external extension texture and each graphics buffer memory block, and storing the binding relationship in a relationship cache set; if it is detected that the non-shared memory of the CPU writes at least one channel of newly added image data, then determining the target graphics buffer memory block and target external extension texture corresponding to each channel of newly added image data; for each channel of newly added image data, running a data writing thread, and writing the newly added image data into the corresponding target graphics buffer memory block. The present application improves the overall synchronization efficiency of multiple channels of image data, shortens the time consumed for overall synchronization update, and greatly reduces the CPU load.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for asynchronously updating multi-channel image data. Background Art

[0002] Assisted driving and autonomous driving technologies are one of the mainstream technologies in the development of smart cars. Among them, assisted driving and autonomous driving technologies usually involve applications such as panoramic imaging and parking. These applications can splice and fuse real-time images collected by multiple cameras into real-time panoramic images and present them to users intuitively to assist users in driving.

[0003] Since panoramic imaging, parking and other applications cannot directly access the camera driver to collect images, they basically obtain camera images from other controllers of the vehicle by pulling streams, or copy image data from other applications in a cross-process manner. This type of image data is usually a fixed-length memory allocated on the CPU (Central Processing Unit).

[0004] In the related technologies, the "serial" synchronous data update method is basically adopted to first copy the image data collected by multiple camera devices to the CPU memory and then synchronously update from the CPU memory to the texture memory of the GPU (Graphics Processing Unit) to splice and fuse the multiple image data into a real-time panoramic image. However, the synchronization process of this "serial" synchronous data update method takes a long time and has a high CPU load. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a method and device for asynchronously updating multiple channels of image data to solve the problem that the synchronization process of the "serial" synchronous data update method in the related art is time-consuming and has a high CPU load.

[0006] According to a first aspect of an embodiment of the present application, a method for asynchronously updating multiple channels of image data is provided, comprising:

[0007] Creating multiple graphics buffer memory blocks and multiple external extension textures; wherein each graphics buffer memory block is set in the CPU to support cross-process memory sharing between the CPU and the GPU; each external extension texture is set in the GPU;

[0008] Based on the application extension interface, a binding relationship between each external extension texture and each graphics buffer memory block is established, and the binding relationship is stored in a relationship cache set;

[0009] If it is detected that at least one channel of newly added image data is written into the non-shared memory of the CPU, a target graphics buffer memory block and a target external extended texture corresponding to each channel of newly added image data are determined;

[0010] For each channel of newly added image data, a data writing thread is run to write the newly added image data into the corresponding target graphics buffer memory block.

[0011] According to a second aspect of an embodiment of the present application, a device for asynchronously updating multiple channels of image data is provided, comprising:

[0012] A creation module is configured to create a plurality of graphics buffer memory blocks and a plurality of external extension textures; wherein each graphics buffer memory block is set in the CPU to support cross-process memory sharing between the CPU and the GPU; each external extension texture is set in the GPU;

[0013] An establishing module configured to establish a binding relationship between each external extended texture and each graphics buffer memory block based on the application extension interface, and store the binding relationship in a relationship cache set;

[0014] The determination module is configured to determine the target graphics buffer memory block and the target external extension texture corresponding to each channel of the newly added image data if it is detected that at least one channel of the newly added image data is written into the non-shared memory of the CPU;

[0015] The writing module is configured to run a data writing thread for each channel of newly added image data, and write the newly added image data into the corresponding target graphics buffer memory block.

[0016] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0017] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0018] Compared with the prior art, the beneficial effects of the embodiment of the present application include at least the following: by creating multiple graphics buffer memory blocks and multiple external extension textures, and establishing a binding relationship between each external extension texture and each graphics buffer memory block based on an application extension interface, when monitoring that at least one new image data is written to the non-shared memory of the CPU, for each new image data, a data writing thread is run to write the new image data to the corresponding target graphics buffer memory block, and the entire data synchronization process only needs to complete the copying of the new image data from the non-shared memory to the target graphics buffer memory block within the CPU, saving the transmission overhead and bandwidth from CPU synchronization to GPU, being more friendly to the overall system performance, and improving the overall synchronization efficiency of multiple image data, shortening the time consumption of the overall synchronization update, and greatly reducing the CPU load. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a method for synchronizing multiple image data in CPU memory to texture memory of GPU;

[0022] Figure 3 It is a flowchart of a method for asynchronously updating multiple channels of image data provided by an embodiment of the present application;

[0023] Figure 4 It is a flowchart of a method for asynchronously updating multiple channels of image data according to an application example;

[0024] Figure 5 It is a structural schematic diagram of a device for asynchronously updating multiple channels of image data provided by an embodiment of the present application;

[0025] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0026] Among them, the reference numerals in the figure are:

[0027] 101-CPU, 102-GPU;

[0028] 501-create module, 502-establish module, 503-determine module, 504-write module;

[0029] 600-Electronic equipment;

[0030] 601 - processor, 602 - memory, 603 - computer program. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0032] A method and device for asynchronously updating multiple channels of image data according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 , the application scenario may include a CPU 101, a GPU 102, and image acquisition devices 1, 2, 3, ..., m. The CPU 101 is provided with non-shared memory and graphic buffer memory blocks 1, 2, 3, ..., n; the GPU 102 is provided with external extended textures 1, 2, 3, ..., k. The image acquisition devices 1, 2, 3, ..., m may be cameras, for example, m surround view cameras arranged on a vehicle (such as a new energy vehicle, etc.). n, m, k are all integers greater than 1, and n≥m, n≥k.

[0034] Applications such as panoramic imaging and parking can run on CPU 101. Applications such as panoramic imaging and parking usually use images collected by more than four surround-view cameras to meet the stitching requirements of panoramic images. In some system architectures, applications such as panoramic imaging and parking cannot directly access and drive the surround-view camera to collect images. Instead, they obtain the image data collected by the surround-view camera from other controllers on the vehicle through streaming, or copy image data from other applications through cross-process methods. This type of image data is usually a fixed-length memory allocated on the CPU.

[0035] OpenGLES (Open Graphics Library for Embedded Systems) is a standard image rendering interface for embedded platforms, which can run on the GPU 102 and use the powerful performance of the GPU to create and render complex graphics effects. It can render 2D and 3D images and models.

[0036] In the related art, when synchronizing image data stored in CPU memory (i.e., non-shared memory in the CPU) to the texture memory of the GPU, the functions glTexImage2D and glTexSubImage2D in OpenGLES are usually used to synchronize multiple image data in the CPU memory to the OpenGLES 2D texture.

[0037] Figure 2 This is a flowchart of a method for synchronizing multiple image data in CPU memory to GPU texture memory. Figure 2 , assuming that m-channel image data collected by m image acquisition devices (such as surround view cameras) in the CPU memory need to be synchronized to the texture memory of the GPU, the synchronization process includes the following steps:

[0038] Step S201, initialize OpenGLES.

[0039] Step S202, initializing multiple 2D textures. When creating a 2D texture during initialization, the image size, format and data type stored in the texture can be set through the glTexImage2D function.

[0040] Step S203, calling the glTexSubImage2D function to update the image data 1 collected by the image acquisition device 1 to the texture memory 1 of the GPU. In one example, an application (such as a panoramic image, a parking application, etc.) can open the image acquisition device 1 by calling an interface. After the image device 1 is opened, it collects and reports the image data 1 to the CPU memory, and then sets the memory address of the image data 1 to OpenGLES for processing. OpenGLES calls the glTexSubImage2D function to update the image data 1 collected by the image acquisition device 1 to the texture memory 1 of the GPU.

[0041] Step S204, calling the glTexSubImage2D function to update the image data 2 acquired by the image acquisition device 2 to the texture memory 2 of the GPU.

[0042] Step S205, calling the glTexSubImage2D function to update the image data 3 acquired by the image acquisition device 3 to the texture memory 3 of the GPU.

[0043] Step S206, calling the glTexSubImage2D function to update the image data m acquired by the image acquisition device m to the texture memory m of the GPU.

[0044] Similarly, referring to the example of step S203 above, an application (such as a panoramic image, a parking application, etc.) can open the image acquisition device 2, 3, ..., m by calling an interface. After the image device 2, 3, ..., m is opened, it acquires and reports the image data 2, 3, ..., m to the CPU memory, and then sets the memory address of the image data 2, 3, ..., m to OpenGLES for processing. OpenGLES calls the glTexSubImage2D function to update the image data 2, 3, ..., m acquired by the image acquisition device 2, 3, ..., m to the texture memory 2, 3, ..., m of the GPU.

[0045] Step S207: drawing.

[0046] In the above synchronization process, steps S203 to S207 are repeatedly (looped) executed. Each time drawing is performed, the texture ID is first bound, and then the glTexSubImage2D function is called to synchronize the image data on the CPU memory to the texture memory of the GPU, and then drawing is performed.

[0047] Step S208, release the texture and release OpenGLES.

[0048] As an example, the pseudo code to initialize a 2D texture is as follows:

[0049] / / Declare an unsigned integer array of length 1

[0050] GLuint textures[1] = {0};

[0051] / / Generate a texture ID and save the generated texture in the passed array textures

[0052] glGenTextures(1, textures);

[0053] / / Bind the texture ID to the 2D texture. After binding, all settings for the 2D texture will take effect on textures[0]

[0054] glBindTexture(GL_TEXTURE_2D, textures[0]);

[0055] / / Set texture wrapping and texture filtering parameters

[0056] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);

[0057] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

[0058] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

[0059] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

[0060] / / Specify a 2D (2-dimensional) texture storage image. The parameters include the image format as RGBA, the image width and height, the texture data type as unsigned bytes, and a pointer to the image data in memory.

[0061] / / After this code is executed, the image format, image width and height, data type of the texture image will be specified, and the data in mImageBuffer will be updated to the texture

[0062] glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, mImageWidth, mImageHeight, 0,GL_RGBA, GL_UNSIGNED_BYTE,mImageBuffer);

[0063] / / Unbind the texture

[0064] glBindTexture(GL_TEXTURE_2D, GL_NONE).

[0065] If multiple channels of image data need to be synchronized (for example, 4 channels of image data need to be synchronized), then each channel of image data needs to be synchronized one after another ("serial" synchronization). The pseudo code is as follows:

[0066] / / Update image data mImageBuffer0 (such as image data 1 captured by image acquisition device 1) to 2D texture textures[0]

[0067] glBindTexture(GL_TEXTURE_2D, textures[0]);

[0068] glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, mImageWidth, mImageHeight,

[0069] GL_RGBA, GL_UNSIGNED_BYTE, mImageBuffer0);

[0070] / / Update image data mImageBuffer1 (such as image data 2 captured by image acquisition device 2) to 2D texture textures[1]

[0071] glBindTexture(GL_TEXTURE_2D, textures[1]);

[0072] glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, mImageWidth, mImageHeight,

[0073] GL_RGBA, GL_UNSIGNED_BYTE, mImageBuffer1);

[0074] / / Update image data mImageBuffer2 (such as image data 3 captured by image acquisition device 3) to 2D texture textures[2]

[0075] glBindTexture(GL_TEXTURE_2D, textures[2]);

[0076] glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, mImageWidth, mImageHeight,

[0077] GL_RGBA, GL_UNSIGNED_BYTE, mImageBuffer2);

[0078] / / Update image data mImageBuffer3 (such as image data 4 captured by image acquisition device 4) to 2D texture textures[3]

[0079] glBindTexture(GL_TEXTURE_2D, textures[3]);

[0080] glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, mImageWidth, mImageHeight,

[0081] GL_RGBA, GL_UNSIGNED_BYTE, mImageBuffer3).

[0082] Since the glTexSubImage2D function must bind the texture first and then synchronize the data, and all operations related to OpenGLES need to be performed in a thread with an EGL context (EGL is the interface between OpenGL ES and the underlying Native platform window system), this means that when multiple image data need to be synchronized, they can only be synchronized "serially".

[0083] In related technologies, the "serial" synchronous update method is used to first copy each channel of image data to the CPU memory and then synchronously update from the CPU memory to the GPU's texture memory. This not only brings a high CPU load, but also the synchronization process is time-consuming. For example, if the application requires a rendering frame rate of 25 frames, then a drawing cycle is 40 milliseconds. Once the data synchronization time is too long, its drawing cycle is likely to exceed 40ms, and its rendering effect will have a noticeable sense of lag, thus affecting the user experience.

[0084] In view of this, the embodiment of the present application proposes a method for asynchronously updating multiple image data, by creating multiple graphics buffer memory blocks and multiple external extension textures, and establishing a binding relationship between each external extension texture and each graphics buffer memory block based on an application extension interface. When it is detected that the non-shared memory of the CPU writes at least one new image data, a data writing thread is run for each new image data, and the new image data is written to the corresponding target graphics buffer memory block. The entire data synchronization process only needs to complete the copying of the new image data from the non-shared memory to the target graphics buffer memory block within the CPU, saving the transmission overhead and bandwidth from the CPU synchronization to the GPU, being more friendly to the overall system performance, and improving the overall synchronization efficiency of multiple image data, shortening the time consumption of the overall synchronization update, and greatly reducing the CPU load. In addition, the GPU can directly read the corresponding target external extension texture from the target graphics buffer memory block based on the above binding relationship, which is convenient for subsequent image drawing, fusion and splicing operations based on the target external extension texture.

[0085] Figure 3 FIG. 1 is a flow chart of a method for asynchronously updating multiple channels of image data provided by an embodiment of the present application. Figure 3 As shown, the method for asynchronously updating multiple channels of image data comprises the following steps:

[0086] Step 301, creating multiple graphics buffer memory blocks and multiple external extension textures; wherein each graphics buffer memory block is set in the CPU to support cross-process memory sharing between the CPU and the GPU; each external extension texture is set in the GPU.

[0087] As an example, applications such as panoramic imaging and parking can create GraphicBuffer components and use the GraphicBuffer components to apply for memory blocks that support cross-process shared memory between the CPU and the GPU, namely, graphic buffer memory blocks, in an operating system (such as an Android operating system). These graphic buffer memory blocks are set in the CPU 101 to cache image data copied from the non-shared memory of the CPU.

[0088] Among them, the GraphicBuffer component is an object used to describe the graphics buffer in Android.

[0089] As an example, when creating a GraphicBuffer component, you need to specify the length and purpose. The pseudo code is as follows:

[0090] / / Declare a structure to save the created GraphicBuffer component

[0091] typedef struct {

[0092] / / Declare a variable to indicate whether the GraphicBuffer component has been written with image data;

[0093] bool isIdle;

[0094] / / Declare a smart pointer variable to save the created GraphicBuffer component

[0095] android::sp <android::graphicbuffer>graphic_buffer_;

[0096] / / Declare a pointer variable to save the buffer address (i.e. local buffer address) used by the GraphicBuffer component to bind to OpenGLES

[0097] void* native_buffer_;

[0098] / / Declare a pointer variable to save the memory address of the GraphicBuffer component mapped to the application process; the image data can be copied to this memory address within the application

[0099] void *virt_addr_[1];

[0100] } Image;

[0101] / / Encapsulate a function to create a GraphicBuffer component. The width and height parameters need to be passed in. The created object, pointer address, etc. are copied to the Image structure.

[0102] void allocGraphicBufferImage(int width, int height, Image* image) {

[0103] / / Define variables to receive the return value of the function

[0104] int ret;

[0105] / / Set the image buffer format to be created to CbYCrY_422_I, which is UYVY

[0106] uint32_t format = HAL_PIXEL_FORMAT_CbYCrY_422_I;

[0107] / / Set the purpose of the created GraphicBuffer component

[0108] / / GRALLOC_USAGE_HW_FB indicates that the memory is allocated in the system frame buffer

[0109] / / GRALLOC_USAGE_HW_RENDER indicates that hardware rendering needs to be supported, that is, OpenGLES is supported to read image data

[0110] / / GRALLOC_USAGE_SW_READ_OFTEN indicates that support for CPU reading data is required

[0111] / / GRALLOC_USAGE_SW_WRITE_OFTEN indicates the need to support CPU writing data

[0112] uint32_t usage = GRALLOC_USAGE_HW_FB | GRALLOC_USAGE_HW_RENDER |

[0113] GRALLOC_USAGE_SW_READ_OFTEN | GRALLOC_USAGE_SW_WRITE_OFTEN;

[0114] / / Create a GraphicBuffer component

[0115] image->graphic_buffer_ = new android::GraphicBuffer(width_, height_,format, 0, usage);

[0116] / / Initialize the GraphicBuffer component

[0117] ret = graphic_buffer_->initCheck();

[0118] / / Get the memory address of the local buffer of the GraphicBuffer component. The type of the local buffer is ANativeWindowBuffer*. You can use the memory address of this local buffer to bind to the OpenGLES texture.

[0119] image->native_buffer_ = graphic_buffer_->getNativeBuffer();

[0120] / / Get the handle of the GraphicBuffer component

[0121] struct private_handle_t *private_hndl;

[0122] private_hndl =

[0123] (struct private_handle_t *)graphic_buffer_->getNativeBuffer()->handle;

[0124] int fd_ = private_hndl->fd;

[0125] / / Map the GraphicBuffer component handle to the current process and return the mapped memory address

[0126] image->virt_addr_[0] = (void *)mmap(nullptr, size_, PROT_READ | PROT_WRITE,

[0127] MAP_SHARED, fd_, 0);

[0128] }.

[0129] In OpenGLES, images are represented by texture memory, so textures need to be created. Furthermore, since image data is imported from outside OpenGLES, the texture type needs to be set to external extended texture (or called external extended texture).

[0130] As an example, applications such as panoramic imaging and parking can use OpenGLES to create external extended textures (such as GL_TEXTURE_EXTERNAL_OES). The pseudo code is as follows:

[0131] / / Declare an unsigned integer array of length 1

[0132] GLuint textures[1] = {0};

[0133] / / Generate a texture ID and save the generated texture in the passed array textures

[0134] glGenTextures(1, textures);

[0135] / / Bind the texture ID to the external extended texture. After binding, all settings for the 2D texture will take effect on textures[0]

[0136] glBindTexture(GL_TEXTURE_EXTERNAL_OES, textures[0]);

[0137] / / Set texture wrapping and texture filtering parameters

[0138] glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);

[0139] glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

[0140] glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

[0141] glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

[0142] / / Unbind the texture

[0143] glBindTexture(GL_TEXTURE_EXTERNAL_OES, GL_NONE).

[0144] Step 302: Based on the application extension interface, a binding relationship between each external extension texture and each graphics buffer memory block is established, and the binding relationship is stored in a relationship cache set.

[0145] Different operating systems use different application extension interfaces. For example, in the Android operating system, the application extension interface can use the EGL extension interface.

[0146] In some examples, one image acquisition device corresponds to at least two graphics buffer memory blocks; one image acquisition device corresponds to one external extended texture. For example, if there are four image acquisition devices, four external extended textures can be set, and the image acquisition devices correspond to the external extended textures one by one; eight graphics buffer memory blocks are set, and each image acquisition device corresponds to two graphics buffer memory blocks.

[0147] Each created graphics buffer memory block is bound to each external extended texture in a one-to-one correspondence to establish a binding relationship between them, and the binding relationship is cached in a relationship cache set, wherein the relationship cache set may be a double-column set.

[0148] Step 303 : if it is detected that at least one channel of newly added image data is written into the non-shared memory of the CPU, a target graphics buffer memory block and a target external extended texture corresponding to each channel of newly added image data are determined.

[0149] Newly added image data refers to new image data acquired by the image acquisition device.

[0150] As an example, when the image acquisition device acquires new image data, it will report it to the CPU, and the CPU will write the new image data reported by the image acquisition device into the non-shared memory of the CPU. When applications such as panoramic imaging and parking detect that at least one new image data is written into the non-shared memory of the CPU, they will search for the source of each new image data, that is, the image acquisition device, and determine the corresponding target graphics buffer memory block and target external extension texture based on the image acquisition device of the new image data.

[0151] Step 304 , for each channel of newly added image data, run a data writing thread to write the newly added image data into the corresponding target graphics buffer memory block.

[0152] As an example, some data writing threads can be created in advance and stored in the standby thread pool. When applications such as panoramic imaging and parking detect that the CPU writes m new image data to the non-shared memory, m data writing threads can be called from the standby thread pool and run to copy and write the m new image data to the corresponding m target graphics buffer memory blocks.

[0153] As another example, when applications such as panoramic imaging and parking detect that m new image data are written to the CPU's non-shared memory, they can instantly create and run m data writing threads to copy and write the m new image data into the corresponding m target graphics buffer memory blocks.

[0154] Compared with the related art, the technical solution provided by the embodiment of the present application only needs to complete the copying of the newly added image data from the non-shared memory to the target graphics buffer memory block within the CPU during the entire data synchronization process, saving the transmission overhead and bandwidth from the CPU synchronization to the GPU, being more friendly to the overall system performance, and improving the overall synchronization efficiency of multi-channel image data, shortening the time consumption of the overall synchronization update, and greatly reducing the CPU load. In addition, the GPU can directly read the corresponding target external extension texture from the target graphics buffer memory block based on the above-mentioned binding relationship, which is convenient for subsequent image drawing, fusion and splicing operations based on the target external extension texture.

[0155] In some embodiments, determining a target graphics buffer memory block corresponding to each channel of newly added image data includes:

[0156] For each channel of newly added image data, determine the image acquisition device corresponding to the newly added image data;

[0157] Determine at least two candidate graphic buffer memory blocks corresponding to the newly added image data based on a preset associated logical relationship between the image acquisition device and the graphic buffer memory block;

[0158] A free graphics buffer memory block is selected from at least two candidate graphics buffer memory blocks as a target graphics buffer memory block.

[0159] The preset logical relationship between the image acquisition device and the graphics buffer memory block may be a mapping relationship established based on the first identification information of the image acquisition device and the second identification information of the graphics buffer memory block. The first identification information and the second identification information may be strings of numbers, words, English, etc. The first identification information is different from the second identification information.

[0160] As an example, the mapping relationship between the image acquisition device and the graphic buffer memory block is shown in Table 1 below.

[0161] Table 1 Mapping relationship between image acquisition device and graphics buffer memory block

[0162]

[0163] In Table 1, Collect_1 represents image acquisition device 1, Mblock_1-1 represents graphics buffer memory block 1-1 corresponding to image acquisition device 1, and Mblock_1-y represents graphics buffer memory block 1-y corresponding to image acquisition device 1. Collect_2 represents image acquisition device 2, Mblock_2-1 represents graphics buffer memory block 2-1 corresponding to image acquisition device 2, and Mblock_2-y represents graphics buffer memory block 2-y corresponding to image acquisition device 2. Collect_m represents image acquisition device m, Mblock_m-1 represents graphics buffer memory block m-1 corresponding to image acquisition device m, and Mblock_m-y represents graphics buffer memory block my corresponding to image acquisition device m.

[0164] As an example, assuming that the panoramic imaging application detects that one new image data is written to the non-shared memory of the CPU, such as the new image data 1 collected and reported by the image acquisition device 1, then the above Table 1 can be queried to obtain at least two candidate graphic buffer memory blocks (Mblock_1-1~Mblock_1-y) corresponding to each image acquisition device 1, and then a free graphic buffer memory block is selected from these candidate graphic buffer memory blocks as the target graphic buffer memory block.

[0165] The free graphics buffer memory block refers to the free frame to which no data is written.

[0166] It can be understood that when multiple channels of newly added image data are written into the non-shared memory of the CPU, their target graphics buffer memory blocks can be determined by referring to the above method.

[0167] In the above implementation mode, each channel of newly added image data corresponds to at least two alternative graphic buffer memory blocks, and the idle graphic buffer memory block is used as the target graphic buffer memory block to cache the newly added image data. Data can be written to and read from different graphic buffer memory blocks alternately, that is, data is not written to and read from the same graphic buffer memory block at the same time, which can avoid problems such as frame tearing and screen distortion.

[0168] In some embodiments, for each channel of newly added image data, a data writing thread is run to write the newly added image data into a corresponding target graphics buffer memory block, including:

[0169] For each channel of newly added image data, run a data writing thread to determine the memory address of the newly added image data in the non-shared memory and its data length;

[0170] Based on the memory address and the data length, the newly added image data is copied from the non-shared memory and written to the target graphics buffer memory block, and the state parameter of the target graphics buffer memory block is set to a non-idle state.

[0171] As an example, when applications such as panoramic imaging and parking detect that at least one new image data is written to the CPU's non-shared memory, they can run a data writing thread for each new image data and call the system's memcpy function to copy the new image data to the corresponding target graphics buffer memory block.

[0172] In one example, the pseudo code for copying the newly added image data to the target graphics buffer memory block is as follows:

[0173] typedef struct {

[0174] / / Declare a variable to indicate whether the newly added image data has been written;

[0175] bool isIdle;

[0176] / / Declare a smart pointer variable to save the created GraphicBuffer component

[0177] android::sp <android::graphicbuffer>graphic_buffer_;

[0178] / / Declare a pointer variable to save the buffer address (i.e. local buffer address) used by the GraphicBuffer component to bind to OpenGLES

[0179] void* native_buffer_;

[0180] / / Declare a pointer variable to save the memory address of the GraphicBuffer component mapped to the application process; the newly added image data can be copied to this memory address within the application

[0181] void *virt_addr_[1];

[0182] } Image;

[0183] / / For each new image data channel, create at least two GraphicBuffer components to synchronize the data;

[0184] Image* imageArray[2];

[0185] / / The function call triggered by the image acquisition device when reporting the newly added image data will pass the memory address (addr) where the newly added image data is saved and the length (len) of the newly added image data

[0186] void onStreamCb(int cameraId, void *addr, size_t len, int frameId,uint64_t timestamp) {

[0187] / / Find an idle GraphicBuffer component (i.e., a target graphics buffer memory block whose state parameter is not idle) from at least two created GraphicBuffer components

[0188] Image* image = nullptr;

[0189] for (int I = 0; i<2; i++) {

[0190] if (imageArray[i]->isIdle) {

[0191] image = imageArray[i];

[0192] break;

[0193] }

[0194] }

[0195] / / If no free frame (i.e., free GraphicBuffer component, or free graphics buffer memory block) is found, it means that the GPU has not finished reading the data in the GraphicBuffer component, so no new image data will be written

[0196] / / Note: To avoid frame tearing, screen distortion and other issues, the application cannot write new image data to the same GraphicBuffer component while the GPU is reading data from it.

[0197] if (nullptr == image) {

[0198] return;

[0199] }

[0200] / / Copy the number of bytes of length len from the memory address addr to the address corresponding to virt_addr[0]

[0201] memcpy(image->virt_addr[0], len, addr);

[0202] / / Change the state parameter of the idle GraphicBuffer component to a non-idle state; the GPU will read data based on this state parameter

[0203] image->isIdle = false;

[0204] / / Remind that the image data has been updated and notify the GPU to refresh the display

[0205] notifyNativeBufferUpdate(image);

[0206] }.

[0207] In some embodiments, establishing a binding relationship between each external extended texture and each graphics buffer memory block includes:

[0208] Get the local buffer address of each graphics buffer memory block;

[0209] For each graphics buffer memory block, create an image object that can be recognized by the application extension interface based on the local buffer address;

[0210] Based on the application extension interface, a binding relationship between the image object and the external extension texture corresponding to the graphics buffer memory block is established.

[0211] As an example, the pseudo code for establishing the binding relationship between each external extended texture and each graphics buffer memory block is as follows:

[0212] / / Declare a two-column set (i.e., relational cache set) to save the binding relationship between nativeBuffer (graphics buffer memory block) and texture (external extended texture);

[0213] / / Only one binding is required, and then the binding relationship is cached; when the corresponding nativeBuffer data changes later, the bound external extension texture is directly taken out from the double column collection for drawing

[0214] std::map<void*, int> mNativeBufferTexMap;

[0215] / / Declare a double column set to save the correspondence between nativeBuffer and the created EGLImageKHR;

[0216] std::map<void*, EGLImageKHR> mNativeBufferEGLImageMap;

[0217] int bindNativeBuffer2OesTex(int ​​tex, void* native_buffer) {

[0218] / / Get the function address of the EGL extension interface

[0219] / / Note: This EGL extension interface is an extended function, which requires the motherboard system to support obtaining the function address through eglGetProcAddress

[0220] / / If a motherboard does not support this extension function, it will return NULL, but now mainstream Android system motherboards all support this extension function

[0221] PFNEGLCREATEIMAGEKHRPROC eglCreateImageKHR;

[0222] eglCreateImageKHR = (PFNEGLCREATEIMAGEKHRPROC) eglGetProcAddress("eglCreateImageKHR");

[0223] / / Get the handle related to the EGL display device

[0224] EGLDisplay dpy = eglGetCurrentDisplay();

[0225] / / Create an image object that can be recognized by the EGL extension interface based on the local buffer address of the GraphicBuffer component

[0226] EGLImageKHR image;

[0227] image = eglCreateImageKHR(dpy, EGL_NO_CONTEXT, EGL_NATIVE_BUFFER_ANDROID, native_buffer, NULL);

[0228] / / Determine whether the creation is successful, if not, exit

[0229] if (image == EGL_NO_IMAGE_KHR) {

[0230] return -1;

[0231] }

[0232] / / Activate texture 0

[0233] glActiveTexture(GL_TEXTURE0);

[0234] / / Bind the passed texture to GL_TEXTURE_EXTERNAL_OES;

[0235] glBindTexture(GL_TEXTURE_EXTERNAL_OES, tex);

[0236] / / Get the EGL extension interface for binding EGL recognizable image objects and external extension textures

[0237] PFNGLEGLIMAGETARGETTEXTURE2DOESPROC glEGLImageTargetTexture2DOES;

[0238] glEGLImageTargetTexture2DOES = (PFNGLEGLIMAGETARGETTEXTURE2DOESPROC)eglGetProcAddress(

[0239] "glEGLImageTargetTexture2DOES");

[0240] / / Bind the EGL-recognizable image object to the external extended texture

[0241] glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, (GLeglImageOES)image);

[0242] / / Unbind the texture

[0243] glBindTexture(GL_TEXTURE_EXTERNAL_OES, GL_NONE);

[0244] / / Deactivate the texture

[0245] glActiveTexture(GL_NONE);

[0246] / / Cache binding relationship

[0247] mNativeBufferTexMap[native_buffer] = tex;

[0248] mNativeBufferEGLImageMap[native_buffer] = image;

[0249] return 0;

[0250] }.

[0251] The above implementation method establishes a binding relationship between each external extended texture and each graphics buffer memory block. When each graphics buffer memory block writes new image data, the external extended texture corresponding to the graphics buffer memory block can be updated synchronously, which makes it easier for the GPU to directly read the external extended texture corresponding to each graphics buffer memory block, thereby ensuring that OpenGLES will not appear stuck when rendering, thereby improving the user experience.

[0252] In some embodiments, the above method may further include the following steps:

[0253] If all the newly added image data has been written into the corresponding target graphics buffer memory block, an image drawing notification is sent to the renderer;

[0254] Based on the image drawing notification, the renderer reads the target external extension texture corresponding to the target graphics buffer memory block from the relationship cache set, and uses the target external extension texture to draw the image.

[0255] As an example, when an application such as panoramic imaging or parking detects that all newly added image data of a certain channel has been written into the corresponding target graphics buffer memory block, an image drawing notification is sent to the OpenGLES renderer, and the image drawing notification includes the local buffer address of the target graphics buffer memory block. When the OpenGLES renderer receives the image drawing notification, it can query the relational cache set according to the local buffer address of the target graphics buffer memory block to find the target external extension texture corresponding to the local buffer address of the target graphics buffer memory block, and then use the target external extension texture to draw the image.

[0256] In one example, the OpenGLES renderer retrieves the bound target external extension texture from the relational cache set based on the value of native_buffer (the local buffer of the GraphicBuffer component), and then directly uses the target external extension texture for drawing. The pseudo code is as follows:

[0257] Functions used by the renderer to process drawing

[0258] void onDrawFrame{

[0259] / / According to the target graphics buffer memory block, when writing the newly added image data, the image->native_buffer passed in takes out the target external extended texture from the double column collection

[0260] int tex = mNativeBufferTexMap[image->native_buffer];

[0261] / / Activate texture 0

[0262] glActiveTexture(GL_TEXTURE0)

[0263] / / Bind texture

[0264] glBindTexture(

[0265] GL_TEXTURE_EXTERNAL_OES,

[0266] tex )

[0268] / / Draw

[0269] glDrawArrays(GL_TRIANGLE_FAN, 0, 4)

[0270] / / After drawing, set the state parameter of the data frame (target graphics buffer memory block) to the idle state; only then will the target graphics buffer memory block be written with new image data

[0271] Image->isIdle = true;

[0272] }.

[0273] The corresponding pseudo code of the fragment shader is as follows:

[0274] / / OpenGLES version number

[0275] #version 300 es

[0276] / / Indicates the use of GL_OES extended function

[0277] #extension GL_OES_EGL_image_external_essl3 : require

[0278] / / Floating point precision is high level;

[0279] precision highp float;

[0280] / / Texture sampler type is samplerExternalOES corresponding to GL_TEXTURE_EXTERNAL_OES

[0281] uniform samplerExternalOES sample_camera;

[0282] / / Texture coordinates, passed in by the CPU; usually {0.0,0.0,0.0,1.0,1.0,1.0,1.0,0.0}

[0283] in vec2 v_texCoord;

[0284] / / Calculate the color value of the fragment to be displayed

[0285] layout(location = 0) out vec4 outColor;

[0286] void main() {

[0287] / / Get the color value from the texture

[0288] vec4 texColor = texture(sample_camera, v_texCoord);

[0289] / / Directly assign values ​​to GPU for display

[0290] outColor = texColor;

[0291] }.

[0292] In the above implementation, after all the newly added image data has been written into the target graphics buffer memory block, the OpenGLES renderer is notified to read the data and draw the image, and then output to the display device for display, which can improve the user experience.

[0293] In some embodiments, the above method further includes:

[0294] If it is determined that the target external extended texture has been read, the state parameter of the target graphics buffer memory block is set to an idle state, and the binding relationship between the target external extended texture and the target graphics buffer memory block is released, and the relationship cache set is refreshed.

[0295] As an example, when an application such as panoramic imaging or parking determines that the GPU has finished reading the target external extended texture, the state parameter of the target graphics buffer memory block is set to an idle state to release the target graphics buffer memory block so that new image data can be written to the target graphics buffer memory block. At the same time, the binding relationship between the target external extended texture and the target graphics buffer memory block is released, and the relationship cache set is refreshed.

[0296] In one example, the pseudo code for releasing the binding relationship between the target external extended texture and the target graphics buffer memory block is as follows:

[0297] The local buffer address of the GraphicBuffer component and the external extended texture need to be passed in for unbinding operations

[0298] mNativeBufferEGLImageMap[native_buffer] = image;

[0299] int unbindNativeBuffer2OesTex(int ​​tex, void* native_buffer) {

[0300] / / Activate texture 0

[0301] glActiveTexture(GL_TEXTURE0);

[0302] / / Bind the passed texture to GL_TEXTURE_EXTERNAL_OES;

[0303] glBindTexture(GL_TEXTURE_EXTERNAL_OES, tex);

[0304] / / Get the EGL extension interface for binding EGL recognizable image objects and external extension textures

[0305] PFNGLEGLIMAGETARGETTEXTURE2DOESPROC glEGLImageTargetTexture2DOES;

[0306] glEGLImageTargetTexture2DOES = (PFNGLEGLIMAGETARGETTEXTURE2DOESPROC)eglGetProcAddress(

[0307] "glEGLImageTargetTexture2DOES");

[0308] / / The EGLImageKHR binding corresponding to the external extended texture is empty (binding empty means unbinding)

[0309] glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, NULL);

[0310] / / Unbind the texture

[0311] glBindTexture(GL_TEXTURE_EXTERNAL_OES, GL_NONE);

[0312] / / Deactivate the texture

[0313] glActiveTexture(GL_NONE);

[0314] PFNEGLDESTROYIMAGEKHRPROC eglDestroyImageKHR;

[0315] eglDestroyImageKHR = (PFNEGLDESTROYIMAGEKHRPROC) eglGetProcAddress("eglDestroyImageKHR");

[0316] EGLDisplay dpy = eglGetCurrentDisplay();

[0317] EGLImageKHR image = mNativeBufferEGLImageMap[native_buffer];

[0318] eglDestroyImageKHR(dpy, image);

[0319] }.

[0320] In some embodiments, after determining that all the newly added image data have been written into the corresponding target graphics buffer memory block, the method further includes:

[0321] Release the data writing thread for writing the newly added image data into the corresponding target graphics buffer memory block, cache the data writing thread into the recycling thread pool, and start recording the recycling time of the data writing thread entering the recycling thread pool;

[0322] If the collection time reaches the preset waiting time and is still not scheduled, the data writing thread will be destroyed.

[0323] The preset waiting time can be flexibly set according to actual conditions. For example, it can be set to 5 minutes, 10 minutes, etc., and the embodiments of the present application do not impose specific restrictions on this.

[0324] As an example, after determining that a certain newly added image has been written into the corresponding target graphics buffer memory block, applications such as panoramic imaging and parking call the thread exit function to release the data writing thread used to write the newly added image data into the corresponding target graphics buffer memory block, recycle the data writing thread into the recycle thread pool, start the timer, and record the recycle time t1 of the data writing thread entering the recycle thread pool. If the data writing thread is still not scheduled for use after the accumulated waiting time (such as 5 minutes) is reached starting from the recycle time t1, the data writing thread is destroyed.

[0325] The above implementation can achieve delayed destruction of data writing threads cached in the thread recycling pool, thereby improving the reuse rate of data writing threads.

[0326] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0327] Figure 4 FIG. 1 is a flow chart of a method for asynchronously updating multiple channels of image data in an application example of the present application. Figure 4 , the method comprises the following steps:

[0328] Step S401, initialize OpenGLES.

[0329] Step S402, initializing multiple external extension textures.

[0330] Step S403: Bind each external extended texture and the local buffer address of each graphics buffer memory block.

[0331] Step S404, searching for bound external extended textures according to the local buffer address of the updated data. According to the local buffer address of the target graphics buffer memory block corresponding to the newly added image data, searching for the external extended texture corresponding to the local buffer address from the relational cache set.

[0332] Step S405: GPU renders the image.

[0333] Step S406, release the texture and release OpenGLES.

[0334] Step S407, creating a graphic buffer memory block. At least two graphic buffer memory blocks corresponding to the image acquisition device 1 may be created in the CPU.

[0335] Step S408, obtaining the local buffer address of each graphics buffer memory block, and mapping the memory in the application process according to the handle of the graphics buffer memory block.

[0336] Step S409, turning on the image acquisition device 1.

[0337] Step S410: the image acquisition device 1 reports the newly added image data.

[0338] Step S411, copying the newly added image data to the corresponding target graphics buffer memory block.

[0339] Step S412, notify OpenGLES to draw.

[0340] Step S413, close the image acquisition device 1 and stop acquiring flow.

[0341] The above steps S407 to S413 are the process of applying to pull the newly added image data from the image acquisition device 1. Similarly, the process of applying to pull the newly added image data from the image acquisition devices 2 to m is the same as the process of applying to pull the newly added image data from the image acquisition device 1.

[0342] During the asynchronous data update process, steps S404 to S405 and steps S410 to S412 are repeatedly executed.

[0343] In order to further illustrate the technical effect of the technical solution provided by the embodiment of the present application, by comparing the technical solution of asynchronously updating multiple channels of image data provided by the embodiment of the present application with the traditional technical solution of serially synchronously updating multiple channels of image data, the test method and test results are as follows:

[0344] Test 1:

[0345] 1) Test environment: Synchronize the 4-way camera data collected by 4 surround view cameras, render at a frame rate of 25fps, and an image resolution of 1280×800.

[0346] 2) Test platform: Qualcomm 6125 Android platform (8 cores, total system CPU load is 800%).

[0347] 3) Test results are as follows: using the technical solution of asynchronously updating multiple channels of image data provided in the embodiment of the present application, the CPU load rate is 21% / 800%, while using the traditional technical solution of serially synchronously updating multiple channels of image data, the CPU load rate is 39% / 800%.

[0348] Test 2:

[0349] 1) Test environment: Synchronize the 4-way camera data collected by 4 surround view cameras, render at a frame rate of 25fps, and an image resolution of 1280×800.

[0350] 2) Test platform: Qualcomm 8155 platform (6 cores are allocated to the Android system, and the total system CPU load is 600%).

[0351] 3) Test results are as follows: using the technical solution of asynchronously updating multiple channels of image data provided in the embodiment of the present application, the CPU load rate is 16% / 800%, while using the traditional technical solution of serially synchronously updating multiple channels of image data, the CPU load rate is 29% / 800%.

[0352] It can be seen from the above that compared with the traditional technical solution of serially and synchronously updating multiple channels of image data, the technical solution of asynchronously updating multiple channels of image data provided in the embodiment of the present application reduces the CPU load by nearly half, which is more friendly to the overall performance of the system.

[0353] In summary, the technical solution provided by the embodiment of the present application not only solves the time-consuming problem of synchronizing multiple image data, ensuring that OpenGLES does not freeze when rendering, but also can greatly reduce the CPU load, which is more friendly to the overall performance of the system.

[0354] The only overhead of the technical solution of the embodiment of the present application is the need to copy the newly added image data from the CPU's non-shared memory to the graphics buffer memory block inside the CPU. Compared with the traditional technical solution of serial synchronous updating of multiple image data, it can save the transmission performance overhead and bandwidth of synchronizing a large amount of data (taking a single-channel 1280x800 UYVY format image as an example, 1.95Mb of data needs to be synchronized at a time) from the CPU memory to the GPU's texture memory, which is more friendly to the overall performance of the system.

[0355] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0356] Figure 5 Schematic diagram of a device for asynchronously updating multiple channels of image data provided by an embodiment of the present application. Figure 5 As shown, the device for asynchronously updating multiple channels of image data includes:

[0357] The creation module 501 is configured to create a plurality of graphics buffer memory blocks and a plurality of external extension textures; wherein each graphics buffer memory block is set in the CPU to support cross-process memory sharing between the CPU and the GPU; each external extension texture is set in the GPU;

[0358] An establishing module 502 is configured to establish a binding relationship between each external extended texture and each graphics buffer memory block based on the application extension interface, and store the binding relationship in a relationship cache set;

[0359] The determination module 503 is configured to determine the target graphics buffer memory block and the target external extension texture corresponding to each channel of the newly added image data if it is detected that at least one channel of the newly added image data is written into the non-shared memory of the CPU;

[0360] The writing module 504 is configured to run a data writing thread for each channel of newly added image data, and write the newly added image data into the corresponding target graphics buffer memory block.

[0361] In some embodiments, the above-mentioned determination module 503 includes:

[0362] A first determining unit is configured to determine, for each channel of newly added image data, an image acquisition device corresponding to the newly added image data;

[0363] A second determining unit is configured to determine at least two candidate graphic buffer memory blocks corresponding to the newly added image data based on a preset associated logical relationship between the image acquisition device and the graphic buffer memory block;

[0364] The screening unit screens out an idle graphics buffer memory block from at least two candidate graphics buffer memory blocks as a target graphics buffer memory block.

[0365] In some embodiments, the writing module 504 includes:

[0366] The running unit is configured to run a data writing thread for each channel of newly added image data to determine the memory address of the newly added image data in the non-shared memory and its data length;

[0367] The writing unit is configured to copy the newly added image data from the non-shared memory and write it into the target graphics buffer memory block based on the memory address and the data length, and set the state parameter of the target graphics buffer memory block to a non-idle state.

[0368] In some embodiments, the establishment module 502 includes:

[0369] An acquisition unit configured to acquire a local buffer address of each graphics buffer memory block;

[0370] The creation unit is configured to create, for each graphics buffer memory block, an image object recognizable by the application extension interface based on the local buffer address;

[0371] The establishing unit is configured to establish a binding relationship between the image object and the external extended texture corresponding to the graphics buffer memory block based on the application extension interface.

[0372] In some embodiments, the above device further includes:

[0373] A notification module configured to send an image drawing notification to the renderer if all the newly added image data has been written into the corresponding target graphics buffer memory block;

[0374] The renderer is configured to read a target external extension texture corresponding to a target graphics buffer memory block from a relational cache set based on an image drawing notification, and use the target external extension texture to draw an image.

[0375] In some embodiments, the above device further comprises:

[0376] The release module is configured to set the state parameter of the target graphics buffer memory block to an idle state if it is determined that the target external extended texture has been read, release the binding relationship between the target external extended texture and the target graphics buffer memory block, and refresh the relationship cache set.

[0377] In some embodiments, the above apparatus further comprises:

[0378] a release module configured to release a data writing thread for executing writing of the newly added image data into the corresponding target graphics buffer memory block when it is determined that all the newly added image data has been written into the corresponding target graphics buffer memory block, cache the data writing thread into a recycling thread pool, and start recording a recycling time when the data writing thread enters the recycling thread pool;

[0379] The destruction module is configured to destroy the data writing thread if the recycling time reaches the preset waiting time and is still not scheduled.

[0380] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0381] Figure 6 is a schematic diagram of an electronic device 600 provided in an embodiment of the present application. Figure 6 As shown, the electronic device 600 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0382] The electronic device 600 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 600 may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that Figure 6 The electronic device 600 is merely an example and does not limit the electronic device 600 . The electronic device 600 may include more or less components than those shown in the figure, or different components.

[0383] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0384] The memory 602 may be an internal storage unit of the electronic device 600, for example, a hard disk or memory of the electronic device 600. The memory 602 may also be an external storage device of the electronic device 600, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 600. The memory 602 may also include both an internal storage unit of the electronic device 600 and an external storage device. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0385] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0386] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electric carrier signals and telecommunication signals.

[0387] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.< / android::graphicbuffer> < / android::graphicbuffer>

Claims

1. A method for asynchronously updating multiple channels of image data, characterized in that: include: Creating multiple graphics buffer memory blocks and multiple external extension textures; wherein each of the graphics buffer memory blocks is set in the CPU to support cross-process memory sharing between the CPU and the GPU; each of the external extension textures is set in the GPU; Based on the application extension interface, establishing a binding relationship between each of the external extension textures and each of the graphics buffer memory blocks, and storing the binding relationship in a relationship cache set; If it is detected that at least one channel of newly added image data is written into the non-shared memory of the CPU, a target graphics buffer memory block and a target external extended texture corresponding to each channel of the newly added image data are determined; For each channel of the newly added image data, running a data writing thread to write the newly added image data into a corresponding target graphics buffer memory block; Based on the application extension interface, a binding relationship between each of the external extension textures and each of the graphics buffer memory blocks is established, including: Obtaining the local buffer address of each of the graphic buffer memory blocks; For each of the graphics buffer memory blocks, based on the local buffer address, create an image object identifiable by an application extension interface; Based on the application extension interface, a binding relationship between the image object and the external extended texture corresponding to the graphics buffer memory block is established.

2. The method according to claim 1, characterized in that Determining the target graphics buffer memory block corresponding to each of the newly added image data includes: For each channel of the newly added image data, determining an image acquisition device corresponding to the newly added image data; Determine at least two candidate graphic buffer memory blocks corresponding to the newly added image data based on a preset associated logical relationship between the image acquisition device and the graphic buffer memory block; An idle graphics buffer memory block is selected from at least two candidate graphics buffer memory blocks as a target graphics buffer memory block.

3. The method according to claim 1, characterized in that For each channel of the newly added image data, a data writing thread is run to write the newly added image data into a corresponding target graphics buffer memory block, including: For each channel of the newly added image data, running a data writing thread to determine the memory address and data length of the newly added image data in the non-shared memory; Based on the memory address and the data length, the newly added image data is copied from the non-shared memory and written into a target graphics buffer memory block, and a state parameter of the target graphics buffer memory block is set to a non-idle state.

4. The method according to claim 1, characterized in that: The method further comprises: If the newly added image data has been written into the corresponding target graphics buffer memory block, an image drawing notification is sent to the renderer; The renderer reads a target external extension texture corresponding to the target graphics buffer memory block from the relational cache set based on the image drawing notification, and uses the target external extension texture to perform image drawing.

5. The method according to claim 4, characterized in that The method further comprises: If it is determined that the target external extended texture has been read, the state parameter of the target graphics buffer memory block is set to an idle state, the binding relationship between the target external extended texture and the target graphics buffer memory block is released, and the relationship cache set is refreshed.

6. The method according to claim 1, characterized in that After determining that the newly added image data has been written into the corresponding target graphics buffer memory block, the method further includes: Release the data writing thread for executing writing of the newly added image data into the corresponding target graphics buffer memory block, cache the data writing thread into a recycling thread pool, and start recording the recycling time of the data writing thread entering the recycling thread pool; If the recycle time reaches the preset waiting time and is still not scheduled, the data writing thread is destroyed.

7. A device for asynchronously updating multiple channels of image data, characterized in that: include: A creation module is configured to create a plurality of graphics buffer memory blocks and a plurality of external extension textures; wherein each of the graphics buffer memory blocks is set in the CPU to support cross-process memory sharing between the CPU and the GPU; each of the external extension textures is set in the GPU; An establishing module, configured to establish a binding relationship between each of the external extended textures and each of the graphics buffer memory blocks based on an application extension interface, and store the binding relationship in a relationship cache set; The determination module is configured to determine the target graphics buffer memory block and the target external extension texture corresponding to each channel of the newly added image data if it is detected that at least one channel of newly added image data is written into the non-shared memory of the CPU; A writing module is configured to run a data writing thread for each channel of the newly added image data, and write the newly added image data into a corresponding target graphics buffer memory block; Based on the application extension interface, a binding relationship between each of the external extension textures and each of the graphics buffer memory blocks is established, including: Obtaining the local buffer address of each of the graphic buffer memory blocks; For each of the graphics buffer memory blocks, based on the local buffer address, create an image object identifiable by an application extension interface; Based on the application extension interface, a binding relationship between the image object and the external extended texture corresponding to the graphics buffer memory block is established.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images