Image display method and system, electronic equipment and storage medium

Abstract

The invention provides an image display method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining and deploying a drawing surface, and creating the drawing surface based on an image reading class in an Android native system; taking the drawing surface as an image rendering surface, and constructing an image rendering environment; synthesizing the received camera data stream into image data based on an image rendering environment; and in response to an image display request of the front end, sending the image data, so that the surface view control of the front end renders the image data on the active page. According to the method, the drawing surface provided by the image reading class creation is bound to the image rendering environment, so that the image rendering environment can operate and synthesize the image data at the rear end, and when the business activity page is started, the surface view control is triggered to directly render the image data to the page, so that the high-performance display of the image is realized, and the loading delay of the image display is avoided.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image display method, system, electronic device and storage medium. Background Art

[0002] OpenGL ES (Open Graphics Library for Embedded Systems) is a standard image rendering interface for embedded platforms. It allows developers to use the powerful performance of GPU (Graphics Processing Unit) to create and render complex graphics effects.

[0003] In related technologies, when using OpenGLES, the system is required to provide an implementation of the EGL (Embedded Graphics Library) interface and a Surface (drawing surface). EGL is the standard interface of OpenGLES, which is implemented by system manufacturers on different platforms and operating systems; Surface can be a buffer memory or a structure representing a screen window. Through OpenGLES, the image display in the active page can be realized, but when entering the active page, there are still problems such as image display loading delay. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present application discloses an image display method, system, electronic device and storage medium, which are used to solve the technical problem of image display loading delay when entering an active page in the prior art.

[0005] In a first aspect, the present application provides an image display method, the method comprising: obtaining a drawing surface and deploying it, the drawing surface being created based on an image reading class in an Android native system; using the drawing surface as an image rendering surface to construct an image rendering environment; synthesizing image data with a received camera data stream based on the image rendering environment; and sending the image data in response to an image display request from a front end, so that a surface view control of the front end renders the image data on an active page.

[0006] In one embodiment of the present application, the drawing surface is used as an image rendering surface to construct an image rendering environment, including: creating a management thread, calling an embedded graphics library interface in the management thread, and establishing a connection with an operating system display service through the embedded graphics library interface; creating a rendering context, and creating an image rendering surface creation class, setting the drawing surface as the image rendering surface; binding the operating system display service, the rendering context and the image rendering surface to generate the image rendering environment.

[0007] In one embodiment of the present application, creating an image rendering surface creation class and setting the drawing surface as the image rendering surface includes: calling the image rendering surface creation class, defining a drawing surface interface corresponding to the drawing surface, the drawing surface interface including a surface creation method; calling the drawing surface interface, executing the surface creation method, and setting the drawing surface as the image rendering surface.

[0008] In one embodiment of the present application, synthesizing image data based on the received camera data stream based on the image rendering environment includes: setting an algorithm renderer and creating a back-end rendering process, wherein the algorithm renderer is configured with a first drawing function; in the back-end rendering process, executing the first drawing function to process the camera data stream in a current buffer; calling a buffer exchange function pre-set in the image rendering environment, submitting the processed camera data stream in the current buffer to the drawing surface, and obtaining the image data.

[0009] After synthesizing the image data with the received camera data stream based on the image rendering environment in one embodiment of the present application, it also includes: in a pre-created scheduling thread, if it is detected that the image data exists in the drawing surface, triggering a callback function to report the image data to an image data object, wherein the image data object is created by the image reading class, and the image data object includes a hardware buffer acquisition function; calling the hardware buffer acquisition function to acquire a hardware buffer, and writing the image data to the hardware buffer, and binding the hardware buffer to a texture memory, so that when the image display request is received, the surface view control reads the image data from the texture memory; if the image display request is not received, continuing the image data synthesis operation until an exit instruction for the image data synthesis operation is received.

[0010] In one embodiment of the present application, so that the surface view control of the front end renders the image data on the active page, including: calling the surface view control, obtaining the image data from the texture memory, the surface view control is provided with an image renderer, and the image renderer is configured with a second drawing function; creating a front-end rendering process, and in the front-end rendering process, executing the second drawing function, obtaining window information of the display window on the active page; rendering the image data to the display window according to the window information.

[0011] In one embodiment of the present application, the method for creating the drawing surface includes: calling the image reading class to create an image reading instance, and setting image data parameters of the image reading instance, wherein the image data parameters include image width, image height, image format and maximum number of images; setting a drawing buffer according to the image data parameters to obtain the drawing surface.

[0012] In a second aspect, the present application provides an image display system, which includes: an acquisition module for acquiring and deploying a drawing surface, wherein the drawing surface is created based on an image reading class in an Android native system; a construction module for using the drawing surface as an image rendering surface to construct an image rendering environment; a synthesis module for synthesizing image data from a received camera data stream based on the image rendering environment; and a display module for sending the image data in response to an image display request from a front end, so that the surface view control of the front end renders the image data on an active page.

[0013] In a third aspect, the present application provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the image display method described in the first aspect.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer executes the image display method described in the first aspect.

[0015] As described above, the image display method, system, electronic device and storage medium provided by the embodiments of the present application have the following beneficial effects: First, the drawing surface is obtained and deployed. The drawing surface is created based on the image reading class in the Android native system. Then, the drawing surface is used as the image rendering surface to build an image rendering environment. Then, the received camera data stream is synthesized with image data based on the image rendering environment. Finally, in response to the image display request of the front end, the image data is sent so that the surface view control of the front end renders the image data on the active page. The drawing surface is created and provided through the image reading class, and the drawing surface is bound to the image rendering environment as the image rendering surface, so that the image rendering surface does not depend on the business activity page, so that the image rendering environment can run in the back end, that is, the algorithm is deployed and the synthesized image is deployed in the back end. When the business activity page is started, that is, when the image data needs to be consumed and displayed, the surface view control can be directly triggered to read the image data synthesized by the back end and directly rendered to the page, so as to complete the image display with high performance, without the need to create a drawing surface for image drawing and rendering through the surface view control after the activity page is started. In this way, the computing task is completed in the back end and the rendering is directly performed in the front end, which avoids loading delays, improves business performance, and improves user experience.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 is a schematic diagram of an implementation environment of an image display system shown in an exemplary embodiment of the present application; Figure 2 is a flowchart of an image display method shown in an exemplary embodiment of the present application; Figure 3 is a flowchart of a method for constructing an image rendering environment shown in an exemplary embodiment of the present application; Figure 4 is a flow chart of a back-end image data synthesis method shown in an exemplary embodiment of the present application; Figure 5 is a flow chart of a front-end image display method shown in an exemplary embodiment of the present application; Figure 6 is a flowchart of a specific image display method shown in an exemplary embodiment of the present application; Figure 7is a block diagram of an image display system shown in an exemplary embodiment of the present application; Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0019] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present application in a schematic manner, and therefore the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0020] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0021] First of all, it should be noted that OpenGLES, as a standard image rendering interface for embedded platforms, relies on the system to provide the implementation of the EGL interface and the provision of Surface. In the field of imaging, OpenGLES has important uses, and many algorithm functions can be implemented based on it. For example, in the vehicle cockpit, the real-time surround image and 3D model fusion function of panoramic parking in the panoramic image can be realized, as well as the watermarking of the driving recorder, real-time encoding-recording into a video file, etc. However, the inventors of this application have found that the current OpenGLES still has the following problems: 1. Only after entering the active page, the component that renders the OpenGLES graphics is added to the window, and its Surface is valid, and OpenGLES can be initialized; 2. OpenGLES needs to be released when exiting the page; 3. When repeatedly entering the page, it is necessary to repeatedly initialize / deinitialize OpenGLES, which is time-consuming and affects the experience; 4. When OpenGLES is in an uninitialized state, the algorithms it runs cannot be used, and it is impossible to do some state settings, resource loading, or back-end running algorithms in advance. As a result, the images synthesized by the back end cannot be displayed with high performance, resulting in loading delays and affecting the experience.

[0022] Therefore, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of an image display system shown in an exemplary embodiment of the present application. Figure 1 As shown, the implementation environment may include an image display system 110 and a computer device 120. The image display system 110 may be set in the computer device 120 to implement an image display function. The computer device 120 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, a neural network computer, etc. The image display system 110 provides a drawing surface through an image reading class creation, and binds the drawing surface to an image rendering environment as an image rendering surface, so that the image rendering surface does not depend on the business activity page, so that the image rendering environment can run in the backend, that is, deploy algorithms and synthesize images in the backend. When the business activity page is started, that is, when the image data needs to be consumed and displayed, the surface view control can be directly triggered to read the image data synthesized in the backend, and directly rendered to the page, so that the image display is completed with high performance, without the need to create a drawing surface through the surface view control for image drawing and rendering after the activity page is started. In this way, the computing task is completed in the backend, and the rendering is directly performed in the frontend, which avoids loading delays, improves business performance, and improves user experience.

[0023] See also Figure 2 , Figure 2is a flowchart of an image display method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the method may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0024] In addition, in the embodiments of the present application, the image display method can be applicable to the Android system, that is, it can be applied to mobile devices using the Android system, such as mobile phones, car computers, etc., and can also be applied to other operating systems.

[0025] On the Android platform, a native control GLSurfaceView (Graphics LayerSurfaceView) is provided, which is mainly used to create and manage the OpenGLES graphics environment. This control inherits from SurfaceView (surface view), encapsulates the relevant drawing environment and interfaces, can draw images and display them directly on the screen, but there are still the following problems: 1. Only after entering the Activity (activity page), the GLSurfaceView is added to the window, and its Surface is valid and OpenGLES can be initialized; 2. OpenGLES must be released when exiting the page; 3. When entering the page repeatedly, OpenGLES needs to be initialized / deinitialized repeatedly; 4. When OpenGLES is not initialized, its running algorithms cannot be used, and it is impossible to do some state settings, resource loading or back-end algorithm running in advance.

[0026] Based on the image display method in the present application, a Surface is created and provided through the image reading class of Android. The drawing surface does not depend on the Activity. The Surface is bound to OpenGLES as the image rendering surface, so that OpenGLES can run on the back end, that is, the algorithm and synthesized image are deployed on the back end. When the Activity is started, that is, when the image data needs to be consumed and displayed, the GLSurfaceView can be directly triggered to read the image data synthesized by the back end and directly rendered to the page, so that the image display is completed with high performance, and there is no need to create a Surface for image drawing and rendering through GLSurfaceView after the Activity is started. In this way, the computing task is completed on the back end and rendering is performed directly on the front end, which avoids loading delays, improves business performance, and improves user experience.

[0027] like Figure 2 As shown, in an exemplary embodiment, the image display method at least includes steps S210 to S240, which are described in detail as follows: Step S210, obtaining and deploying a drawing surface, where the drawing surface is created based on an image reading class in the Android native system.

[0028] In the native Android system, GLSurfaceView uses the Surface receiving algorithm of the window in the active page to process and output the image for display. The control is bound to the life cycle of the active page. The Surface is created when the active page is entered and becomes invalid when the Surface is exited. The image reading class Imagereader in the native Android system provides image reading functions and can provide a Surface that is not dependent on the backend of the active page. When the content in the Surface is updated, there will be a corresponding callback function to notify. Imagereader also supports reading image data in the Surface.

[0029] In this embodiment, considering that the Surface of GLSurfaceView depends on the active page, it is impossible to pre-load the algorithm for data processing before the active page pops up, resulting in a delay in image display loading. Therefore, ImageReader is used to provide Surface and deployed on the back end. OpenGLE is pre-initialized and the algorithm is loaded for image processing before entering the page. In this way, high-performance display of the front-end image can be guaranteed.

[0030] In one embodiment, a method for creating a drawing surface includes: calling an image reading class to create an image reading instance, and setting image data parameters of the image reading instance, where the image data parameters include image width, image height, image format, and maximum number of images; setting a drawing buffer according to the image data parameters to obtain a drawing surface.

[0031] In this embodiment, when ImageReader is used, it will create an image reading instance, that is, set the width, height, format and maxImages (maximum number of images) of the image it receives. Based on the above image data parameters, a corresponding buffer, that is, Surface, can be set to receive the image data. In this way, it is ensured that the subsequent image data is processed in the expected manner, which improves the efficiency of image processing while ensuring the accuracy of image processing.

[0032] For example, create an ImageReader instance, specify its width as 1920, height as 980, format as RGBX_8888, and support a maximum of 5 images. Then the image data in the Surface will have a width of 1920, a height of 980, and a format of RGBX_8888.

[0033] In another embodiment, when ImageReader is used, it is also necessary to set a frame-ready callback function setOnImageAvailableListener. When there is image data in Surface, there will be a notification in the callback function, so that the front end can obtain the image data in Surface through ImageReader.

[0034] In this embodiment, a listener is created, a listening event of frame ready (i.e., synthesis of a frame of image data is completed) is set, and a corresponding callback function is configured to implement the callback of the image data in the Surface.

[0035] Step S220: Using the drawing surface as an image rendering surface to construct an image rendering environment.

[0036] In this embodiment, the Surface created by ImageReader and independent of the active page is bound to OpenGLE to construct an OpenGLE environment (i.e., an image rendering environment). In this way, the OpenGLE environment can complete the synthesis of image data in advance without relying on the front-end active page.

[0037] In one embodiment, a drawing surface is used as an image rendering surface to construct an image rendering environment, including: creating a management thread, calling an embedded graphics library interface in the management thread, and establishing a connection with an operating system display service through the embedded graphics library interface; creating a rendering context, and creating an image rendering surface creation class, setting the drawing surface as an image rendering surface; binding the operating system display service, the rendering context and the image rendering surface to generate an image rendering environment.

[0038] The creation of an OpenGL ES environment includes at least the following steps: obtaining EGL and operating system display services, initializing EGL and establishing a connection between EGL and the operating system display service, querying and selecting the type and configuration of the available image rendering surface, creating a rendering context, creating an image rendering surface, and binding the operating system display service, the rendering context, and the image rendering surface.

[0039] In this embodiment, an image rendering surface creation class is created, Surface is set as the image rendering surface, and the binding of Surface and OpenGLES environment is realized by implementing a module for managing OpenGLES environment and drawing interface, and the creation of back-end OpenGLES environment is completed. In this way, the OpenGLE environment can complete the synthesis of image data in advance without relying on the front-end active page.

[0040] In another embodiment, the backend is designed with a multi-threaded architecture, including a management thread, a backend rendering thread, and a scheduling thread, wherein the management thread is used to manage the creation of an ImageReader instance, obtain a Surface, and manage the creation, initialization, start, and stop of an OpenGLES environment, the backend rendering thread is used to run an algorithm to synthesize image data and exchange buffers, and the scheduling thread is used to call back image data. In this way, the system stability is improved through reasonable task division and collaborative work.

[0041] In one embodiment, an image rendering surface creation class is created and a drawing surface is set as the image rendering surface, including: calling the image rendering surface creation class, defining a drawing surface interface corresponding to the drawing surface, the drawing surface interface including a surface creation method; calling the drawing surface interface, executing the surface creation method, and setting the drawing surface as the image rendering surface.

[0042] In this embodiment, a new image rendering surface creation class is created to implement the interface for creating OpenGLES Surface, and a function for creating OpenGLES Surface is defined, i.e., the surface creation method. The key parameter is reader.getSurface(), which will be called in OpenGLES to obtain the Surface created by ImageReader and bind it as the image rendering surface. In this way, by creating a new image rendering surface creation class to implement the interface for creating OpenGLESSurface, defining a function for creating OpenGLES Surface, and completing the binding of Surface and OpenGLES, the created OpenGLE environment can pre-complete the synthesis of image data without relying on the front-end active page.

[0043] In another embodiment, the image rendering surface creation class further implements an interface for destroying the OpenGLES Surface and defines a surface destruction method. When the synthesized image data is not needed, the surface destruction method is called and the OpenGLES environment is released to release resources.

[0044] See also Figure 3 , Figure 3 FIG. 1 is a flowchart of a method for constructing an image rendering environment shown in an exemplary embodiment of the present application. Figure 3As shown, the image rendering environment construction method includes how to create an ImageReader instance to obtain Surface and set it to the OpenGLES environment. The image rendering environment construction method at least includes the following steps: using ImageReader to provide Surface, that is, creating an ImageReader instance, setting image data parameters, that is, the width, height, format and maximum number of the image; creating a listening callback scheduler ImageAvailableHandler, and creating a listener OnImageAvailableListener and performing listening events and corresponding callback function onImageAvailable configuration, triggering the callback of image data through the listener receiving the frame ready event, receiving the callback event from the listener through the listening callback scheduler, when synthesizing image data in the OpenGLES environment, the callback function onImageAvailable is called, so that the ImageReader instance obtains the image data, and when the front end enters the active page, if the image needs to be displayed, the image data is read from the ImageReader instance, and the image renderer is called using GLSurfaceView to render the image data; setting the drawing buffer according to the image data parameters to obtain the Surface; creating the implementation ImageReaderWindow of the OpenGLES Surface SurfaceFactory(), that is, create an image rendering surface creation class, bind the Surface created by the ImageReader instance to OpenGLES; use the Surface created by the ImageReader instance as the image rendering surface to create an OpenGLES environment.

[0045] Step S230 , synthesizing the received camera data stream into image data based on the image rendering environment.

[0046] Among them, the image rendering surface of the image rendering environment is the Surface created by the ImageReader instance. Therefore, the image rendering environment can deploy algorithms to synthesize image data in the back end. In this way, the OpenGLES environment in the back end can load resources in advance, perform algorithm fusion, encoding, recording and other tasks in the back end, support off-screen operation of synthetic image data, and ensure the response speed of the front-end image display.

[0047] In one embodiment, the received camera data stream is synthesized into image data based on the image rendering environment, including: setting an algorithm renderer and creating a backend rendering process, wherein the algorithm renderer is configured with a first drawing function; in the backend rendering process, executing the first drawing function to process the camera data stream in the current buffer; calling a buffer exchange function pre-set in the image rendering environment, submitting the processed camera data stream in the current buffer to the drawing surface, and obtaining image data.

[0048] In this embodiment, after the image rendering environment is started, the internal implementation is to start a backend rendering thread, in which the drawing function of the algorithm renderer is called, the buffer is exchanged after the renderer is executed, and the image data is obtained in the Surface, so that efficient processing of the camera data stream is achieved.

[0049] After the buffers are exchanged, the callback function onImageAvailable in the ImageReader instance is triggered to obtain the image data.

[0050] In one embodiment, after synthesizing the image data with the received camera data stream based on the image rendering environment, it also includes: in a pre-created scheduling thread, if it is detected that there is image data in the drawing surface, triggering a callback function to report the image data to an image data object, the image data object is created by an image reading class, and the image data object includes a hardware buffer acquisition function; calling the hardware buffer acquisition function to acquire the hardware buffer, and writing the image data to the hardware buffer, and binding the hardware buffer to the texture memory, so that when an image display request is received, the surface view control reads the image data from the texture memory; if no image display request is received, continuing the image data synthesis operation until an exit instruction for the image data synthesis operation is received.

[0051] In this embodiment, the frame ready event is listened to by the listener in the scheduling thread, and the callback function is triggered to call back the image data, wherein the image data is reported to the Image object created in the ImageReader instance. The Image object corresponds to a structure AhardwareBuffer, which is a hardware buffer. Then, by binding the hardware buffer to the texture memory, GLSurfaceView can directly read the image data from the texture memory and render it to the active page. In this way, the front end achieves the purpose of rendering the image data synthesized by the back end with high performance.

[0052] Among them, the hardware buffer AhardwareBuffer is an advanced buffer management mechanism used to efficiently share and transmit image data between different components. It provides direct access to the underlying hardware buffer, can share image data between the backend process and the frontend process, and can achieve zero-copy synchronization of image data, so that GLSurfaceView can directly read image data from the texture memory and render it to the active page, reducing the overhead of data transmission and improving the response speed of image display.

[0053] The Image object has a hardware buffer acquisition function Image.getHardwareBuffer(), which can obtain the HardwareBuffer component object. It has a corresponding structure AhardwareBuffer at the bottom of the system. The memory pointed to can be used as shared memory between GPU (Graphics Processing Unit) and CPU (Central Processing Unit). Just bind the GPU texture memory and AhardwareBuffer to achieve zero-copy data synchronization. For GLSurfaceView, an image renderer ImageRender is set to render image data. When the image needs to be consumed, ImageRender directly reads the image data for rendering.

[0054] In addition, the front end sends an image display request, indicating that the active page is opened. At this time, the front end has image data consumption needs and obtains the latest image data through GLSurfaceView to realize image display. If no image display request is received, it indicates that the active page is exited and the front end has no image data consumption needs. At this time, the back end continues to perform image data synthesis operations until it receives an exit instruction for the image data synthesis operation, that is, the corresponding business exits.

[0055] See also Figure 4 , Figure 4 FIG. 1 is a flowchart of a back-end image data synthesis method shown in an exemplary embodiment of the present application. Figure 4As shown, an image data synthesis method under a back-end multi-threaded architecture is presented, and the back-end image data synthesis method includes at least the following steps: in a management thread, an OpenGLES environment is created, a Surface created by ImageReader is bound, and then the OpenGLES environment is initialized, and after the algorithm renderer is set and the first drawing function onDrawFrame() is configured, the OpenGLES environment is started; after the OpenGLES environment is started, the internal implementation is to start the back-end rendering process, in the back-end rendering process, the algorithm function onDrawFrame() is called to synthesize image data, and then the buffer exchange function eglSwapBuffers is called to submit the image data to the Surface, and it is determined whether the algorithm exits, if not, the image data synthesis operation is continued, and if so, the OpenGLES environment is released in the management thread, wherein the determination of whether the algorithm exits is determined by whether the business exits, and if it exits, the OpenGLES environment is directly stopped, it is determined that the algorithm needs to exit, and then the OpenGLES environment is released; and when exchanging buffers, an image data callback is triggered, that is, in the scheduling thread, the ImageReader callback function onImageAvailable is triggered, so as to obtain the synthesized image data, and the image data can be displayed or processed again when necessary.

[0056] In this way, if the business is not exited, the backend OpenGLES environment will not be released and will remain in a running state. There is no need to repeatedly initialize or release resources each time entering and exiting the activity page. When switching between different activity pages (including between different interfaces and between full screen and floating windows), it only involves using different GLSurfaceView to display images. There is no need to re-initialize / de-initialize, which will not affect the backend running algorithm and improve the response speed of image display.

[0057] Step S240, in response to the image display request of the front end, sending the image data so that the surface view control of the front end renders the image data on the active page.

[0058] In this embodiment, while supporting the off-screen operation of the OpenGLES environment, it is also compatible with the original GLSurfaceView function, allowing the image synthesized by the algorithm to be displayed on the active page.

[0059] In addition, after the front-end GLSurfaceView finishes rendering and displaying, it needs to return the Image object to ImageReader to continue synthesizing new image data.

[0060] In one embodiment, the surface view control of the front end is used to render image data on the active page, including: calling the surface view control to obtain image data from the texture memory, wherein an image renderer is provided in the surface view control, and a second drawing function is configured in the image renderer; creating a front-end rendering process, and executing the second drawing function in the front-end rendering process to obtain window information of the display window on the active page; and rendering the image data to the display window according to the window information.

[0061] The texture memory is mainly used in the GPU to store texture data for rendering graphics, and has the characteristics of high-speed access. In this embodiment, the texture memory is bound to the aforementioned hardware buffer and stores image data, so GLSurfaceView can obtain image data from the texture memory.

[0062] In this embodiment, an image renderer ImageRender is set in GLSurfaceView, and a second drawing function onDrawFrame is configured in ImageRender to realize the rendering of image data in the front-end rendering process. For example, the GLSurfaceView object on the layout file can be obtained through ID (Identification). In addition, the rendering mode can be set to manual triggering, that is, manually triggering rendering again when a frame of data is received. In the front-end rendering process, it is necessary to render according to the window information of the display window corresponding to the image, such as the position, height, width, etc. of the display area.

[0063] In this way, by calling GLSurfaceView to obtain image data from texture memory, the time and overhead of data transmission are reduced, the rendering efficiency is improved, and the second drawing function in the image renderer is used to perform image rendering according to window information in the front-end rendering process, thereby achieving efficient and accurate display of image data on the window.

[0064] In a possible embodiment, AhardwareBuffer and GPU texture are bound and the GPU texture is drawn. After the front end retrieves the image data, AhardwareBuffer and GPU texture need to be unbound.

[0065] In a possible embodiment, the step of the Image object obtaining the system bottom layer AhardwareBuffer at least includes: calling a hardware buffer acquisition function to obtain the memory address of the AhardwareBuffer at the bottom layer of the HardwareBuffer object, and obtaining AhardwareBuffer according to the address.

[0066] In a possible embodiment, the step of binding AhardwareBuffer to texture memory at least includes: obtaining the address of the extended interface function, obtaining the EGLClientBuffer object from AhardwareBuffer; creating an EGL-recognizable image object EGLImageKHR based on the EGLClientBuffer object obtained from AhardwareBuffer; activating texture unit 0, and binding the incoming texture ID as a 2D texture, binding the EGLImageKHR image object and texture, returning the created image object EGLImageKHR, and completing the binding. In addition, the image object EGLImageKHR needs to be destroyed when releasing resources to avoid memory leaks.

[0067] In a possible embodiment, the unbinding of AhardwareBuffer from the texture memory is achieved by calling a preset unbinding interface, and the binding of AhardwareBuffer from the texture memory is unbinded through the unbinding interface.

[0068] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of a front-end image display method shown in an exemplary embodiment of the present application. Figure 5 As shown, the steps of the front-end image display method at least include: after the back-end synthesizes the image data, triggering the ImageReader callback function onImageAvailable to obtain the image data; at the front-end, if entering the active page, initializing GLSurfaceView and setting the image renderer ImageRender, at this time, setting the state to indicate that the image data needs to be consumed, and notifying the back-end, and starting the front-end rendering thread at the same time, initializing rendering-related resources through the onSurfaceCreated function; when the back-end determines that the image data needs to be displayed, the latest frame of image data is given to the front-end image renderer; at the front-end, the image renderer renders the image data through the onDrawFrame function and displays it attentively; if a page exit operation is detected, the front-end rendering thread is stopped, and at the same time as exiting the active page, the rendering-related resources are released through the onSurfaceRelease function; finally, setting the state to indicate that the image data does not need to be consumed, at the back-end, it is determined that the synthesized image data does not need to be processed, and the image data continues to be synthesized, and the judgment is made after the next frame of image data is synthesized.

[0069] See also Figure 6 , Figure 6 FIG. 1 is a flowchart of a specific image display method shown in an exemplary embodiment of the present application. Figure 6As shown, the specific image display method at least includes: at the back end, initializing the image reading component to provide a drawing surface; initializing the OpenGLES environment running on the back end; initializing the algorithm for synthesizing image data; running the algorithm according to the setting state (that is, when the setting state represents the need to consume image data), and continuously updating the input camera data stream, model, etc. to synthesize image data; at the front end, if entering the active page, the native control GLSurfaceView takes effect; setting and initializing the image renderer; receiving the image data synthesized by the algorithm and rendering it to the window; if it is necessary to exit the active page, releasing the image renderer; then exiting the active page, the native control GLSurfaceView becomes invalid; at the back end, if the business exits, releasing the algorithm; then releasing the OpenGLES environment running on the back end.

[0070] Therefore, in the embodiment of the present application, two sets of OpenGLES environments and two Surfaces are included. On the back end, an OpenGLES environment is created, and a Surface created by ImageReader is bound to support the back end to run OpenGLES. This set of OpenGLES environment and Surface is equivalent to the producer of image data. The core is to deploy algorithms, synthesize image data, and output the synthesized image data to ImageReader, regardless of whether the generated image data is consumed; on the front end, Android native GLSurfaceView is used to display the image data in ImageReader. At this time, GLSurfaceView is equivalent to the front end OpenGLES environment, and the image display window is equivalent to a Surface on the front end. This set of OpenGLES environment and Surface is equivalent to consumers. When display is required, the latest image data will be read from the back end for display.

[0071] The above-mentioned image display method first obtains and deploys a drawing surface, which is created based on the image reading class in the Android native system. Then, the drawing surface is used as an image rendering surface to build an image rendering environment. Then, the received camera data stream is synthesized into image data based on the image rendering environment. Finally, in response to the image display request of the front end, the image data is sent so that the surface view control of the front end renders the image data on the active page. The drawing surface is created and provided through the image reading class, and the drawing surface is bound to the image rendering environment as the image rendering surface, so that the image rendering surface does not depend on the business activity page, so that the image rendering environment can run on the back end, that is, the algorithm is deployed and the image is synthesized on the back end. When the business activity page is started, that is, when the image data needs to be consumed and displayed, the surface view control can be directly triggered to read the image data synthesized by the back end and directly rendered on the page, so as to complete the image display with high performance, without the need to create a drawing surface for image drawing and rendering through the surface view control after the activity page is started. In this way, the computing task is completed on the back end and the rendering is directly performed on the front end, thereby avoiding loading delays, improving business performance, and improving user experience.

[0072] See also Figure 7 , Figure 7 is a block diagram of an image display system shown in an exemplary embodiment of the present application. The system can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the system may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the system is applicable.

[0073] In addition, in the embodiment of the present application, the image display system can be applicable to the Android system, that is, it can be applied to mobile devices of the Android system, such as mobile phones, car terminals, etc.

[0074] like Figure 7 As shown, in an exemplary embodiment, the image display system 700 includes at least a drawing surface acquisition module 710, a construction module 720, a synthesis module 730 and a display module 740, which are described in detail as follows: An acquisition module 710 is used to acquire and deploy a drawing surface, where the drawing surface is created based on an image reading class in the Android native system; A construction module 720 is used to use the drawing surface as an image rendering surface to construct an image rendering environment; A synthesis module 730, configured to synthesize image data from the received camera data stream based on an image rendering environment; The display module 740 is used to send image data in response to the image display request of the front end, so that the surface view control of the front end renders the image data on the active page.

[0075] It should be noted that the image display system provided in the above embodiment and the image display method provided in the above embodiment belong to the same concept, wherein the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0076] See also Figure 8 , Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application. Figure 8 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0077] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803, such as executing the method in the above embodiment. In the RAM 803, various programs and data required for system operation are also stored. The CPU 801, the ROM 802 and the RAM 803 are connected to each other through the bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0078] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0079] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, various functions defined in the system of the present application are executed.

[0080] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the image display method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0081] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0082] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0084] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An image display method, characterized in that: The method comprises: Obtain and deploy a drawing surface, where the drawing surface is created based on an image reading class in the Android native system; Using the drawing surface as an image rendering surface to construct an image rendering environment; synthesizing image data from the received camera data stream based on the image rendering environment; In response to an image display request of a front end, the image data is sent so that a surface view control of the front end renders the image data on an active page.

2. The image display method according to claim 1, characterized in that: The step of using the drawing surface as an image rendering surface and constructing an image rendering environment comprises: Creating a management thread, calling an embedded graphics library interface in the management thread, and establishing a connection with an operating system display service through the embedded graphics library interface; Create a rendering context, create an image rendering surface creation class, and set the drawing surface as the image rendering surface; The operating system display service, the rendering context and the image rendering surface are bound to generate the image rendering environment.

3. The image display method according to claim 2, characterized in that: The creating an image rendering surface creation class, setting the drawing surface as the image rendering surface, comprises: Calling the image rendering surface creation class to define a drawing surface interface corresponding to the drawing surface, wherein the drawing surface interface includes a surface creation method; The drawing surface interface is called, the surface creation method is executed, and the drawing surface is set as the image rendering surface.

4. The image display method according to claim 1, characterized in that: The step of synthesizing the received camera data stream into image data based on the image rendering environment includes: Setting an algorithm renderer and creating a backend rendering process, wherein the algorithm renderer is configured with a first drawing function; In the backend rendering process, executing the first drawing function, and processing the camera data stream in the current buffer; A buffer exchange function pre-set in the image rendering environment is called to submit the processed camera data stream in the current buffer to the drawing surface to obtain the image data.

5. The image display method according to claim 4, characterized in that: After synthesizing the received camera data stream into image data based on the image rendering environment, the method further includes: In a pre-created scheduling thread, if it is detected that the image data exists in the drawing surface, a callback function is triggered to report the image data to an image data object, where the image data object is created by the image reading class and includes a hardware buffer acquisition function; Calling the hardware buffer acquisition function to acquire a hardware buffer, and writing the image data into the hardware buffer, and binding the hardware buffer to a texture memory, so that when receiving the image display request, the surface view control reads the image data from the texture memory; If the image display request is not received, the image data synthesis operation is continued until an exit instruction of the image data synthesis operation is received.

6. The image display method according to claim 5, characterized in that: The method of causing the surface view control of the front end to render the image data on the active page includes: Calling the surface view control to obtain the image data from the texture memory, wherein the surface view control is provided with an image renderer, and the image renderer is configured with a second drawing function; Creating a front-end rendering process, and executing the second drawing function in the front-end rendering process to obtain window information of a display window on the active page; The image data is rendered onto the display window according to the window information.

7. The image display method according to any one of claims 1 to 6, characterized in that: The method of creating the drawing surface includes: Call the image reading class to create an image reading instance, and set image data parameters of the image reading instance, wherein the image data parameters include image width, image height, image format and maximum number of images; A drawing buffer is set according to the image data parameters to obtain the drawing surface.

8. An image display system, characterized in that: The system comprises: An acquisition module, used to acquire and deploy a drawing surface, wherein the drawing surface is created based on an image reading class in the Android native system; A construction module, used to use the drawing surface as an image rendering surface to construct an image rendering environment; A synthesis module, configured to synthesize image data from the received camera data stream based on the image rendering environment; The display module is used to send the image data in response to the image display request of the front end, so that the surface view control of the front end renders the image data on the active page.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the image display method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the image display method as claimed in any one of claims 1 to 7.

Images