Shooting preview method and electronic device

Abstract

The embodiments of the present application relate to the field of terminal technology, and in particular to a shooting preview method and electronic device, which can provide users with high-speed and smooth HDR display effects based on the GPU in the electronic device. The method is applied to an electronic device, the electronic device includes a camera and a display screen, and the electronic device is installed with a camera APP. The method includes: capturing a first image, the first image is a preview image for displaying on the preview interface of the camera APP; processing the first image to obtain a second image; wherein the second dynamic imaging range of the second image is higher than the first dynamic imaging range of the first image; calling the GPU to render the first layer of the second image, performing layer synthesis to obtain and display the preview interface of the camera APP including the second image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to a shooting preview method and an electronic device. Background Art

[0002] Most electronic devices support high dynamic range (HDR) imaging technology, which can record a larger range of brightness information and show more details of bright and dark parts in the image.

[0003] The electronic device may be installed with a camera application (application, APP). The camera APP provides preview, photo taking and video recording functions. Conventional technology cannot present HDR display effects in the preview interface of the camera APP. Among them, the HDR display effect can, for example, make the bright areas in the image brighter, the dark areas darker, the dynamic imaging range of the image wider, the details of the image more obvious, and the colors clearer and brighter. Summary of the invention

[0004] The embodiments of the present application provide a shooting preview method and an electronic device, which can provide a user with a high-speed and smooth HDR display effect based on a graphics processing unit (GPU) in the electronic device.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a shooting preview method is provided, which is applied to an electronic device, wherein the electronic device includes a camera and a display screen, and a camera APP is installed in the electronic device. The method includes: firstly, capturing a first image, wherein the first image is a preview image for displaying on a preview interface of the camera APP; secondly, processing the first image to obtain a second image, wherein a second dynamic imaging range of the second image is higher than a first dynamic imaging range of the first image; finally, calling a GPU to render a first layer of the second image, performing layer synthesis to obtain and display a preview interface of the camera APP including the second image.

[0007] In the present application, the electronic device can capture a first image in real time, and process the first image to obtain a second image. Among them, the first image is a preview image for displaying on the preview interface of the camera APP. After processing, the dynamic imaging range of the second image is higher than the dynamic imaging range of the first image. Therefore, the processed second image has an HDR display effect. Afterwards, the electronic device can call the GPU to render the first layer of the second image. The electronic device can also perform layer synthesis and display the processed second image on the preview interface of the camera APP. In this way, a preview display of the HDR display effect can be achieved, providing users with a high-speed and smooth preview display of the HDR display effect.

[0008] In a possible implementation of the first aspect, after capturing the first image, the electronic device may further identify whether the shooting scene of the first image is a high dynamic imaging range shooting scene. If the shooting scene of the first image is a high dynamic imaging range shooting scene, the first image is processed to obtain a second image. And the second image is displayed on the preview interface of the camera APP. If the shooting scene of the first image is not a shooting scene of a high dynamic imaging range, the electronic device may not process the first image and directly display the first image on the preview interface of the camera APP. In this way, the electronic device can process the first image according to the situation, which helps to save the power consumption of the electronic device.

[0009] In a possible implementation of the first aspect, the electronic device can also identify whether the high dynamic range imaging HDR function is turned on. If the high dynamic range imaging HDR function of the electronic device is turned on, the first image is processed to obtain the second image, and the second image is displayed on the preview interface of the camera APP. If the high dynamic range imaging HDR function of the electronic device is not turned on, the first image is not processed, and the first image is displayed on the preview interface of the camera APP. That is, the electronic device can process the first image according to the activation state of the high dynamic range imaging HDR function, which helps to save power consumption of the electronic device.

[0010] In a possible implementation of the first aspect, the electronic device calls a GPU to render the first layer of the second image through a standard graphics library application programming interface OpenGL API. This implementation provides a possible way for an electronic device to call a GPU.

[0011] In a possible implementation of the first aspect, the electronic device uses a first LUT to process a first image to obtain a second image. The first LUT is used to map the RGB values ​​of some or all pixels in the first image. This implementation provides a possible way for an electronic device to process a first image to obtain a second image. The first LUT may be pre-stored in the electronic device. Alternatively, the electronic device may obtain the first LUT based on the brightness map information of the first image and the screen brightness value of the display screen; wherein the brightness map information includes a histogram corresponding to the first image, or a brightness map corresponding to the first image.

[0012] In a possible implementation of the first aspect, the electronic device includes a first renderer; the camera APP triggers the first renderer to process the first image to obtain the second image. The camera APP triggers the first renderer to call the GPU to render the first layer of the second image. In the present application, the camera APP can trigger the first renderer to process the first image and call the GPU to render the first layer of the second image.

[0013] In a possible implementation of the first aspect, the electronic device includes a first renderer and a second renderer; the camera APP triggers the second renderer to call the GPU to render the second layer of the first image off-screen, and the electronic device does not display the rendered second layer. Afterwards, the camera APP triggers the first renderer to process the rendered first image into the second image. Finally, the camera APP triggers the first renderer to call the GPU to render the first layer of the second image. In the present application, the camera APP may first trigger the second renderer to render the first image, and then trigger the first renderer to process the rendered first image and call the GPU to render the first layer of the second image. In this way, the processing rate of the first renderer can be accelerated, which helps to display the second image with HDR display effect in real time on the preview interface of the camera APP of the electronic device.

[0014] In a possible implementation of the first aspect, the camera APP may first load the embedded system image library EGL environment. Afterwards, the camera APP obtains the first image through the surface texture object. The camera APP triggers the first renderer, calls the image enhancement algorithm, obtains the first LUT, and uses the first LUT to map and process the first image to obtain the second image. The camera APP establishes a correspondence between the first surface object of the second image and the first texture through the surface texture object. Based on the embedded system image library EGL statement, the camera APP triggers the first renderer to call the GPU to render the first texture through the standard graphics library application programming interface OpenGL API to obtain a first layer including the second image. This implementation method provides a possible implementation method for processing the first image and rendering the second image.

[0015] In a possible implementation of the first aspect, the camera APP may first load the embedded system image library EGL environment. The camera APP obtains the first image through the surface texture object. The camera APP establishes a correspondence between the first surface object of the first image and the first texture through the surface texture object. Based on the embedded system image library EGL statement, the camera APP triggers the second renderer to call the GPU to render the first and second textures off-screen through the standard graphics library application programming interface OpenGL API to obtain the second layer of the first image; wherein the second layer is cached in the first frame buffer object FBO. The camera APP triggers the first renderer, calls the image enhancement algorithm, obtains the first LUT, and uses the first LUT to process the first image in the first frame buffer object FBO into the second image. Based on the embedded system image library EGL statement, the camera APP triggers the second renderer to call the GPU to render the second image in the first frame buffer object FBO through the standard graphics library application programming interface OpenGL API to obtain the first layer of the second image. This implementation provides another possible implementation of processing the first image and rendering the second image.

[0016] In a possible implementation of the first aspect, the electronic device receives a first operation of the user, and the first operation is used to open a camera APP. In response to the first operation, a first image is captured. That is, the electronic device can execute a shooting preview method provided by the present application in response to a trigger of the user.

[0017] In a second aspect, the present application provides an electronic device, which includes: a memory, a display screen, a camera, and one or more processors. The camera, the memory, and the display screen are coupled to the processor. The camera is used to capture images, and the display screen is used to display an interface, wherein the memory is used to store computer program codes, and the computer program codes include computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method as in the first aspect and any possible design thereof.

[0018] In a third aspect, the present application provides a chip system, which can be applied to an electronic device including a memory. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The interface circuit is used to receive a signal from the above-mentioned memory and send the signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device executes the method as in the first aspect and any possible design thereof.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.

[0020] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and any possible design thereof.

[0021] It can be understood that the beneficial effects that can be achieved by the electronic device of any possible design of the second aspect, the chip system of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect can be referred to as the beneficial effects in the first aspect and any possible design of the electronic device, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the software and hardware architecture of an electronic device provided in an embodiment of the present application;

[0024] Figure 3 A flowchart of a shooting preview method provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of an interface of an electronic device provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the interaction between various modules in an electronic device provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the architecture of a camera system in an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In digital images, dynamic range can be used to represent the difference between the darkest and brightest in an image. For example, dynamic range can be the ratio of the maximum brightness to the minimum brightness in an image, and its unit can be dB.

[0029] According to different dynamic imaging ranges, images can be divided into standard dynamic range (SDR) images and high dynamic range (HDR) images. For example, the dynamic imaging range of SDR images is between 1 nit and 100 nits. The dynamic imaging range of HDR images is between 0.001 nits and 10,000 nits. Among them, nit is a unit of brightness. It can be seen that compared with ordinary images (such as SDR images), HDR images have expanded the brightness range of the image, so they can record a larger brightness range information and show more bright and dark details in the image.

[0030] In conventional solutions, electronic devices can perform algorithm enhancement processing based on the collected SDR images, so that the processed SDR images have HDR display effects. Among them, the display effects of HDR can be, for example, brighter bright areas in the image, darker dark areas, wider dynamic imaging range of the image, more obvious image details, clearer and brighter colors, etc. Exemplarily, the electronic device can brighten the bright areas in the collected SDR images, darken the dark areas, and highlight the bright areas in the SDR images, so that they have HDR display effects.

[0031] The process of displaying an image by an electronic device includes collecting an image, rendering an image, and displaying an image. The electronic device may include a central processing unit (CPU) and / or a graphics processing unit (GPU). The electronic device may render an image by means of a CPU or a GPU. The GPU has higher performance and efficiency in both parallel computing and image processing, and therefore, more and more electronic devices are rendering images based on GPUs. Exemplarily, the electronic device includes an open graphics library (OpenGL) and an embedded-system graphics library (EGL). Among them, OpenGL is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D images. EGL is an intermediate interface layer between OpenGL and an application. In other words, EGL is a bridge between an application and OpenGL, and only through the EGL application can the GPU be called through OpenGL to achieve image rendering.

[0032] Electronic devices can be installed with a camera application (application, APP). The camera APP provides preview, photo and video recording functions. However, conventional technologies cannot present HDR display effects in the preview interface of the camera APP. Specifically, electronic devices cannot render the image displayed on the preview interface of the camera APP in real time based on the GPU, providing users with high-speed and smooth preview effects while presenting HDR display effects.

[0033] To this end, an embodiment of the present application provides a shooting preview method and an electronic device, wherein the electronic device can capture a first image in real time, and process the first image to obtain a second image. Among them, the first image is a preview image for displaying on the preview interface of the camera APP. After processing, the dynamic imaging range of the second image is higher than the dynamic imaging range of the first image. Therefore, the processed second image has an HDR display effect. Afterwards, the electronic device can call the GPU to render the first layer of the second image, and the electronic device can also perform layer synthesis and display the processed second image on the preview interface of the camera APP. In this way, the electronic device can realize real-time preview display of the HDR display effect, and provide users with a high-speed and smooth preview display of the HDR display effect.

[0034] The method provided in the embodiment of the present application can be applied to electronic devices with data processing capabilities and display screens. The above-mentioned first electronic device may include a server, a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (for example, a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (for example, a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiment of the present application does not impose any restrictions on this. In the embodiment of the present application, the above-mentioned electronic device is an electronic device that can run an operating system and install application programs. Optionally, the operating system running on the electronic device may be system, system, System, etc.

[0035] For example, please refer to Figure 1, which shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, an audio module 130, a speaker 130A, a microphone 130B, a display screen 140, a communication module 150, a power module 160, an input device 170, a sensor module 180, a camera 190, etc. Among them, the sensor module 180 may include a pressure sensor, a touch sensor, etc.

[0036] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0037] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110.

[0038] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0039] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0040] The internal memory 121 may be used to store one or more computer programs, which include instructions. The processor 110 may enable the electronic device 100 to execute the application running method provided in some embodiments of the present application, as well as various applications and data management, by running the above instructions stored in the internal memory 121. In some embodiments, the processor 110 may enable the electronic device 100 to execute the application running method provided in the embodiments of the present application, as well as other applications and data management, by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor 110.

[0041] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 130, the speaker 130A, the microphone 130B, and the application processor. The speaker 130A, also called a "speaker", is used to convert audio electrical signals into sound signals.

[0042] The microphone 130B, also called a "microphone" or "speaker", is used to convert a sound signal into an electrical signal. A user can make a sound by approaching the microphone 130B with his or her mouth, and input the sound signal into the microphone 130B.

[0043] The communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2 and the communication module 150.

[0044] The communication module 150 can provide solutions for wireless communications including cellular, Wi-Fi, Bluetooth (BT), wireless data transmission modules (e.g., 433MHz, 868MHz, 915MHz) applied to the electronic device 100. The communication module 150 can be one or more devices integrating at least one communication processing module. The communication module 150 receives electromagnetic waves via antenna 1 or antenna 2, filters and frequency modulates the electromagnetic wave signals, and sends the processed signals to the processor 110. The communication module 150 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through antenna 1 or antenna 2.

[0045] The electronic device 100 implements the display function through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0046] The display screen 140 is used to display images, videos, etc. The display screen 140 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 140, where N is a positive integer greater than 1. In the embodiment of the present application, the display screen 140 may be used to display a UI and receive user operations on the UI.

[0047] The power module 160 can be used to supply power to various components included in the electronic device 100. In some embodiments, the power module 160 can be a battery, such as a rechargeable battery.

[0048] The input device 170 may include a keyboard, a mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc. into the electronic device 100, thereby issuing commands to the electronic device 100, inputting data, etc.

[0049] The electronic device 100 can realize the shooting function through an ISP, a camera 190, a video codec, a GPU, a display screen 140, and an application processor. The ISP is used to process the data fed back by the camera 190. The camera 190 is used to capture static images or videos. In some embodiments, the electronic device 100 includes 1 or N cameras 190, where N is a positive integer greater than 1. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 290 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc. The video codec is used to compress or decompress digital video.

[0050] Take the above electronic device 100 as a mobile phone. The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 100. The software structure is as follows: Figure 2 shown.

[0051] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into six layers, from top to bottom: application layer, application framework layer, system library, hardware abstraction layer, kernel layer, and hardware.

[0052] The application layer may include a series of application packages. Figure 2 In the embodiment, the electronic device includes a camera APP.

[0053] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a camera service, a graphics synthesizer (Surface Flinger), a brightness management module, a first renderer, and a second renderer. Alternatively, the application framework layer may only include the first renderer.

[0054] The first renderer is used to perform image enhancement on the first image so that it has an HDR display effect, and call the GPU to render the first layer of the second image.

[0055] Alternatively, the second renderer is used to call the GPU to render the second layer of the first image captured by the camera off-screen to obtain the rendered second layer. The electronic device does not display the second layer. Afterwards, the first renderer can perform image enhancement on the first image in the second layer so that it has an HDR display effect, and call the GPU to render the first layer of the second image.

[0056] The brightness management module can respond to the instruction of the camera APP, obtain the screen brightness value of the current display screen, and return the screen brightness value to the camera APP.

[0057] System libraries include the open graphics library (OpenGL).

[0058] The hardware abstraction layer can include a camera hardware abstraction module and a hardware synthesis HWC. The camera APP can control the underlying device to capture the first image in response to the user's first operation. For example, the camera application can start the camera service (Camera Service) by calling a preset API. The camera service can interact with the camera hardware abstraction module (Camera HAL) in the hardware abstraction layer (HAL) during operation. Among them, the camera hardware abstraction module is responsible for interacting with the hardware device (such as a camera) that implements the shooting function in the mobile phone to realize the collection of image data.

[0059] Hardware synthesis HWC provides hardware support for SurfaceFlinger. The camera preview interface can include multiple layers, and SurfaceFlinger can synthesize these multiple layers into a preview interface. Alternatively, SurfaceFlinger can synthesize these multiple layers into a preview interface through hardware synthesis HWC.

[0060] The kernel layer includes the camera driver and the display driver. The hardware may include the camera, display, and GPU.

[0061] As an example, the camera APP instructs the camera to capture the first image in response to the user's first operation. The camera can return each frame of the captured first image to the camera APP through the camera driver, the camera hardware abstraction module, and the camera service.

[0062] The first renderer in the camera app obtains the histogram of the first image. The first renderer calls the image enhancement algorithm according to the first image, the screen brightness value, and the histogram of the first image to obtain a first mapping table, i.e., a first LUT. The first mapping table can represent the mapping logic of the RGB values ​​before and after pixel mapping, and is used to map the RGB values ​​of pixels in the image. The first renderer maps the first image based on the first mapping table to obtain a second image after image enhancement.

[0063] Optionally, the second renderer in the camera APP calls the GPU to render the second layer of the first image off-screen. Afterwards, the first renderer in the camera APP processes the first image in the second layer to obtain the second image after image enhancement.

[0064] Finally, the first renderer calls the GPU to render the first layer of the second image. The SurfaceFlinger or the hardware synthesis HWC synthesizes the layer to obtain a preview interface. Finally, the display screen can display the preview interface.

[0065] The following takes the mobile phone as an example. Figure 3 A shooting preview method and electronic device provided in an embodiment of the present application are introduced.

[0066] S301, the mobile phone collects a first image.

[0067] The first image is a preview image for displaying on a preview interface of the camera APP.

[0068] Exemplarily, the mobile phone may receive a first operation of the user, and in response to the first operation, collect a first image. The first operation is used to open the camera APP. For example, the first operation may be that the user clicks on the application icon of the camera APP on the main interface of the mobile phone. The first operation may also be, for example, a first voice operation. For example, the user speaks a first voice command into the microphone of the mobile phone, and the first voice command may be, for example, "open the camera". In this way, after the user opens the camera APP, the mobile phone can execute S301-S304 and display a preview interface, which includes the second image.

[0069] After the mobile phone captures the first image, it can execute S302 to process the first image. Alternatively, after the mobile phone captures the first image, it can not process the first image first. After recognizing that the high dynamic range HDR function of the mobile phone is turned on, the mobile phone executes S302 to process the first image. If the high dynamic range HDR function of the mobile phone is not turned on, the mobile phone may not process the first image and directly display the first image on the preview interface of the camera APP.

[0070] For example, Figure 4 As shown, the preview interface 401 of the camera APP includes an "HDR" control 402. In response to the user clicking the "HDR" control 402 in the preview interface 401 of the camera APP, the high dynamic range imaging HDR function of the mobile phone is turned on, and the mobile phone can execute S302.

[0071] Optionally, after the mobile phone captures the first image, the mobile phone can identify whether the shooting scene of the first image is a high dynamic range shooting scene (HDR shooting scene). If the shooting scene of the first image is an HDR shooting scene, the mobile phone executes S302 to process the first image. If the shooting scene of the first image is not an HDR shooting scene, the mobile phone may not process the first image and directly display the first image on the preview interface of the camera APP.

[0072] For example, in response to a user clicking on the application icon of the camera APP on the main interface of the mobile phone, the mobile phone can display the collected image in the SDR display mode. The mobile phone can automatically identify the current shooting scene. If the current shooting scene is an HDR scene, the mobile phone switches to the HDR display mode. Exemplarily, in response to the user opening the camera APP, the mobile phone collects a first image, and based on the first image, identifies whether the current shooting scene is an HDR shooting scene. Specifically, for the first image, the mobile phone can count the average contrast of the first image, the average brightness of the dark area in the first image, or the average brightness of the bright area in the first image. If the average contrast of the first image is greater than a preset threshold, or the average brightness of the dark area in the first image is greater than a preset threshold, or the average brightness of the bright area in the first image is greater than a preset threshold, the mobile phone identifies the current shooting scene as an HDR shooting scene. As another example, in response to the user opening the camera APP, the mobile phone collects the first image, and the mobile phone identifies whether the first image includes a preset shooting object. If the first image includes a preset shooting object, the mobile phone identifies the current shooting scene as an HDR shooting scene. The preset shooting object may include preset shooting objects such as the sun, the sky, street lights, and lights. Correspondingly, the HDR shooting scene may be a sunset scene, a sunset scene, and the like.

[0073] S302: The mobile phone performs a first processing on the first image to obtain a second image corresponding to the first image.

[0074] That is, the mobile phone processes the first image to obtain the second image. The second dynamic imaging range of the second image is higher than the first dynamic imaging range of the first image. Among them, the first processing can be image enhancement processing, and the image enhancement processing is used to modify the RGB values ​​of some or all pixels in the first image so that it has an HDR display effect. Taking the HDR display effect of brighter parts and darker parts as an example, the image enhancement processing can be to modify the RGB values ​​of the pixels in the bright area so that the brightness values ​​of the pixels in the bright area are increased, and to modify the RGB values ​​of the pixels in the dark area so that the brightness values ​​of the pixels in the dark area are reduced. For example, the value of one or more of the R component, G component, and B component of the pixel can be modified to modify the brightness value of the pixel.

[0075] The mobile phone may map the RGB values ​​of some or all pixels in the first image based on the first mapping table to obtain the second image.

[0076] Among them, the first mapping table can characterize the mapping logic of the RGB values ​​before and after the pixel mapping. Through the first mapping table, a set of RGB values ​​of the pixel can be mapped to another set of RGB values, thereby changing the brightness of the pixels in the image. In the second image after mapping by the first mapping table, the brightness values ​​of the pixels in the first image whose brightness values ​​exceed the first threshold are higher, and the brightness values ​​of the pixels in the first image whose brightness values ​​are lower than the second threshold are lower. Among them, the first threshold is greater than the second threshold. The pixels whose brightness values ​​exceed the first threshold are located in the bright area of ​​the first image, and the bright area can be, for example, a light source in the first image. The light source can be, for example, the sun, lights, the sky, the moon, etc. The pixels whose brightness values ​​are lower than the second threshold are located in the dark area of ​​the first image. The first threshold and the second threshold can be the same or different.

[0077] Optionally, the first mapping table may be, for example, a first 3D color lookup table (LUT). Exemplarily, the first LUT may be as shown in Table 1.

[0078] Table 1

[0079]

[0080] When the original RGB value is (24, 21, 23), after the mapping of the LUT shown in Table 1, the output RGB value is (7, 9, 10). When the original RGB value is (51, 44, 68), after the mapping of the LUT shown in Table 1, the output RGB value is (67, 27, 49). When the original RGB value is (84, 11, 161), after the mapping of the LUT shown in Table 1, the output RGB value is (107, 87, 197). When the original RGB value is (231, 214, 221), after the mapping of the LUT shown in Table 1, the output RGB value is (255, 247, 243).

[0081] The mobile phone uses the first image and the first mapping table to map the R, G, and B components of each pixel in the first image to obtain the R', G', and B' components of each pixel after mapping, thereby obtaining the second image.

[0082] The first mapping table may be pre-stored in the mobile phone, and the mobile phone may directly call the first mapping table to process the first image.

[0083] Furthermore, the mobile phone can calculate the first mapping table based on the image enhancement algorithm. Exemplarily, the mobile phone inputs the first image, the brightness information of the first image and the current screen brightness value of the display screen into the image enhancement algorithm, and the image enhancement algorithm can output a first mapping table matching the first image. The brightness information of the first image can be a first brightness map corresponding to the first image, or a histogram corresponding to the first image.

[0084] Exemplarily, for example, the brightness value of pixel a in the first image is Y=t1*R+t2*G+t3*14B. Exemplarily, t1, t2, and t3 can be 0.299, 0.587, and 0.114, respectively. R, G, and B are the values ​​of the R component, the G component, and the B component of pixel A, respectively. The mobile phone can calculate the brightness value of each pixel in the first image based on the above calculation method. Then, the mobile phone can obtain the first brightness map corresponding to the first image. In a digital image, the first brightness map can be a matrix, and each element in the matrix corresponds to the brightness value of each pixel in the first image. The mobile phone can also obtain the histogram of the first image. The histogram describes the statistical characteristics of the brightness values ​​of the pixels in the first image. The histogram graphically represents the number of pixels at each brightness level of the first image, and shows the distribution of pixels with different brightness values ​​in the first image.

[0085] S303: The mobile phone calls the GPU to render the first layer of the second image.

[0086] Rendering may include vertex shading, shape assembly, geometry shading, rasterization, fragment shading, testing and blending. The present application embodiment does not specifically limit the rendering, and any existing rendering method may be referred to. The layer rendered by the GPU can be displayed on the display screen.

[0087] The mobile phone calls the GPU in the mobile phone to render the first layer of the second image. Exemplarily, the mobile phone includes an open image library, which may be OpenGL, for example. OpenGL is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D images. EGL is an intermediate interface layer between OpenGL and the application. In other words, EGL is a bridge between the application and OpenGL, and the application can only render the image by calling the GPU through OpenGL based on EGL.

[0088] Exemplarily, in response to the first operation of the user, the mobile phone first loads the EGL environment. Loading the EGL environment includes establishing a connection between the EGL and the display screen, initializing the EGL, loading the EGL configuration, and the like.

[0089] Afterwards, the mobile phone can create a surface texture object, and the surface texture object can obtain each frame of the first image captured by the camera. The surface texture object can obtain each frame of the first image from the memory buffer. The first image is placed in the corresponding first surface object. Afterwards, the mobile phone can process the first image in the first surface object into a second image. The surface texture object can establish a correspondence between the first surface object and the first texture available to OpenGL. Afterwards, OpenGL obtains the first texture according to the texture ID and renders the first image in the first texture. Among them, texture is a term in OpenGL, indicating rendering data that can be recognized and used by OpenGL. The first texture includes image data of the first image. The texture can be cached in a texture buffer. The texture includes a texture ID, and the texture ID is used to uniquely identify this texture. OpenGL renders the image data in the corresponding texture based on the texture ID.

[0090] Optionally, the mobile phone may first call the GPU to render the second layer of the first image off-screen, and then the mobile phone may perform the first processing on the rendered first image in the second layer to obtain the second image. The mobile phone does not display the second layer. Thereafter, the mobile phone calls the GPU to render the first layer of the second image off-screen.

[0091] S304, the mobile phone displays a preview interface including the second image camera APP.

[0092] The display effect of the second image is an HDR display effect. Exemplarily, compared with the first image, the bright areas in the second image are brighter, the dark areas are darker, the dynamic imaging range is wider, the details are more obvious, and the colors are clearer and brighter.

[0093] In the above embodiments, the method of the embodiment of the present application is introduced from the perspective of the mobile phone as the execution subject. The specific implementation of the method provided in the embodiment of the present application requires the support of various software and hardware modules in the mobile phone. In the following embodiments, the method of the embodiment of the present application is introduced through the interaction of various software and hardware modules in the mobile phone.

[0094] For example, Figure 5As shown, the mobile phone may include a camera APP, a camera service (Camera Serivce), a camera hardware abstraction module (Camera HAL), a camera driver (Camera Driver), a camera, and a display screen. Among them, the camera APP may be included in the application layer of the mobile phone. The camera service may be included in the application framework layer of the mobile phone. The camera hardware abstraction module may be included in the hardware abstraction layer of the mobile phone. The camera driver may be included in the kernel layer of the mobile phone. The camera is the hardware of the mobile phone. The camera includes an image sensor, and the image sensor is used to collect RAW images. The mobile phone may also include a graphics synthesizer (surface flinger), which is included in the application framework layer of the mobile phone.

[0095] S1, the camera APP initializes the EGL environment.

[0096] The camera APP can initialize the EGL environment in response to the user's first operation. Among them, initializing the EGL environment includes obtaining a display window. Among them, obtaining the display window is to obtain the identifier of the display screen used to display the image. Initializing the EGL environment also includes establishing a rendering area EGLSurface, associating an OpenGL renderer, etc. Among them, the rendering area is used to cache the second image rendered by the GPU. The renderer is a set of software codes for performing graphics rendering. In an embodiment of the present application, initializing EGL includes initializing a first renderer and a second renderer. Among them, the second renderer can be, for example, a set of codes that call the GPU to render the first image. The second renderer can be, for example, a set of codes that enhance the first image and call the GPU to render the second image. The first renderer and the second renderer can be software modules. Initializing EGL can also include the camera APP creating a corresponding surface texture object.

[0097] Optionally, the camera APP may also create a memory buffer in response to the first operation, where the memory buffer is used to store each frame of the first image captured by the camera.

[0098] Afterwards, the camera APP may send a first request to the camera service. The first request is used to request the camera to capture a first image. In response to the first request, the camera service triggers the camera hardware abstraction module to continuously and periodically capture the first image through the camera.

[0099] S2. The camera APP sends a first request to the camera service.

[0100] Optionally, the first request may include an identifier of a preview mode. Among them, the camera APP may include a photo mode, a preview mode, and a video mode. There is a mapping relationship between the user's input and the preview mode. The camera APP can respond to different user inputs and send photo requests including different modes to the camera service. For example, in the implementation of this application, the first operation corresponds to the preview mode. In response to the user's first operation, the camera APP sends a first request including a preview mode identifier to the camera service, instructing the capture of the first image.

[0101] For example, in response to a user clicking on the application icon of the camera APP on the main interface of the mobile phone, the camera APP sends a first request to the camera service, and the first request includes an identifier of the preview mode. Optionally, the first request may also include information such as the resolution of the image and the frame rate of the display screen.

[0102] S3. In response to receiving the first request, the camera service sends the first request to the camera hardware abstraction module, the camera driver and the camera head in sequence.

[0103] The camera hardware abstraction module may call a camera driver in the kernel layer according to the received first request, and the camera driver drives the camera to collect image data in response to the first request.

[0104] S4. The camera continues to periodically capture the first image in response to the first request.

[0105] The first image may be, for example, a RAW image captured by a camera.

[0106] S5. The camera returns the collected first image to the camera hardware abstraction module through the camera driver.

[0107] S6, in response to receiving the first image, the camera hardware abstraction module places the first image in a memory buffer.

[0108] Before executing S6, the camera hardware abstraction module may pre-process the captured first image. For example, Figure 6 As shown, the camera hardware abstract module includes two data streams. One of them is the data stream of preview data, and the other is the data stream of photo data. The camera hardware abstract module drives the camera to collect the first image based on the first request of the camera service. Specifically, the camera hardware abstract module may include an image front end module (Image Front End, IFE) and an image processing engine (Image processing engine, IPE). IFE will perform image processing such as color correction, downsampling, and de-mosaicing on the RAW image. Afterwards, IPE will continue to perform image processing such as image cropping, noise reduction, color processing, and detail enhancement on the image to obtain the first image.

[0109] Optionally, the camera hardware abstraction module can also drive the camera to collect a RAW image based on the camera service's photo taking instruction. Specifically, the camera hardware abstraction module can also include a Bayer processing segment (BPS). BPS is used for bad pixel removal, phase focusing, de-mosaicing, downsampling, HDR processing, and Bayer hybrid noise reduction processing of image data. Afterwards, IPE will continue to perform image processing such as image cropping, noise reduction, color processing, detail enhancement, etc. on the image to obtain a photo.

[0110] S7, the camera app gets the first image of each frame from the memory buffer.

[0111] Exemplarily, the camera APP obtains each frame of the first image from the memory buffer through the surface texture object. The surface texture object is a class used to display the first image captured by the camera on the preview interface. The surface texture object can establish a corresponding relationship with multiple surface objects, wherein a surface object includes the image data of a first image.

[0112] Exemplarily, the surface texture object can place each frame of the first image reported by the camera hardware abstraction module in the corresponding first surface object. The surface texture object arranges multiple first surface objects in a queue. Afterwards, the surface texture object will establish a correspondence between the first surface object including the first image and the first texture. Among them, texture is a term in OpenGL, indicating a structure that can be recognized and used by OpenGL. The first texture includes image data of the first image. Each first texture includes a corresponding texture ID. Afterwards, the camera APP triggers the second renderer through the EGL instruction to call the image enhancement algorithm based on the texture ID, obtain the first LUT, and use the first LUT to process the first image to obtain the second image.

[0113] In one possible implementation, the camera APP may execute S8a-S8b.

[0114] S8a, the camera APP triggers the first renderer to process the first image to obtain the second image.

[0115] S8b, the camera APP triggers the first renderer to call the GPU to render the first layer of the second image.

[0116] Exemplarily, the camera APP can trigger the first renderer to call the image enhancement algorithm based on the EGL statement, obtain the first LUT, and use the first LUT to process the first image to obtain the second image. Specifically, the first renderer can first obtain the first image in the first texture based on the texture ID. Afterwards, the first renderer can obtain the histogram of the first image. Exemplarily, the first renderer can call the histogram algorithm to obtain the histogram of the first image. The first renderer inputs the first image, the histogram of the first image, and the display brightness value of the display screen into the image enhancement algorithm to obtain the first LUT. Afterwards, the first renderer maps the RGB values ​​of each pixel in the first image based on the first LUT to obtain the mapped second image. Afterwards, the first renderer calls the GPU to render the first texture based on the texture ID to obtain the rendered second image.

[0117] Optionally, the camera APP first calls the second renderer to render the first image, but does not display the first image, and then processes the rendered first image to obtain the second image. Finally, the camera APP calls the first renderer to render the first layer of the second image and displays the second image. Exemplarily, the camera APP may execute S9a-S9c.

[0118] S9a, the camera APP triggers the second renderer to render the second layer of the first image.

[0119] Exemplarily, the camera APP can trigger the second renderer to render the second layer of the first image based on the EGL statement. The second renderer can obtain the first texture including the first image based on the texture ID of the first texture, and then the second renderer can call the GPU to render the first texture to obtain the second layer including the first image. This rendering process can be called off-screen rendering (off-screen-texture), wherein off-screen rendering can be a rendering operation performed in a memory buffer outside the buffer area of ​​the display screen. The first texture includes a corresponding first frame buffer object (framebuffer object, FBO). The second layer rendered by the second renderer can be stored in the first frame buffer object (framebuffer object, FBO), wherein the second layer includes the processed first image.

[0120] Optionally, before calling the GPU to render the first image, the camera APP may pre-process the first image, such as image rotation.

[0121] S9b, the camera APP triggers the first renderer to process the first image in the first frame buffer object to obtain the second image.

[0122] S9c, the camera APP triggers the first renderer to call the GPU to render the first layer of the second image.

[0123] Exemplarily, the camera APP can trigger the second renderer to call the image enhancement algorithm based on the EGL statement, obtain the first LUT, and use the first LUT to process the first to obtain the second image.

[0124] Specifically, the first renderer may obtain a histogram of the first image. Exemplarily, the first renderer may call a histogram algorithm to obtain a histogram of the first image. The first renderer inputs the first image, the histogram of the first image, and the display brightness value of the display screen into an image enhancement algorithm to obtain a first LUT.

[0125] Optionally, the camera hardware abstraction module may obtain a histogram of the first image and send the histogram to the first renderer.

[0126] Afterwards, the first renderer can first obtain the corresponding first FBO based on the first texture ID, and read the image data of the first image in the first FBO. The first renderer maps the RGB values ​​of each pixel in the first image based on the first LUT to obtain the mapped second image. That is, the first image in the first FBO is processed into the second image. Finally, the second renderer calls the GPU to render the second image in the first FBO corresponding to the first texture based on the texture ID of the first texture, and renders to obtain the first layer of the second image.

[0127] That is, the camera APP may first trigger the second renderer to render the first image, then trigger the first renderer to process the first image rendered by the second renderer to obtain the second image, and then render the second image. Alternatively, the camera APP may directly trigger the first renderer to process the first image to obtain the second image, and then render the second image.

[0128] S10, the camera app triggers the surface flinger to synthesize the preview interface and display it.

[0129] The preview interface includes the second image. Exemplarily, the camera APP triggers the surface flinger through the surface view interface to synthesize the preview interface including the second image.

[0130] It should be understood that the display interface of a mobile phone is usually synthesized from at least one layer. Figure 4 As shown, the preview interface 401 may include, for example, a first layer, the first layer including a second image. The first interface may also include a third layer, the third layer including a background image and interface elements of the camera APP, the interface elements including various buttons, such as a photo button, a portrait button, a setting button, etc. Optionally, the preview interface may also include a system image layer, the system image layer including system interface elements. Taking the preview interface including the first layer and the third layer as an example, the camera APP may also call the GPU or CPU to render the third layer.

[0131] Surface flinger is used to synthesize layers, for example, synthesize the first layer and the third layer to obtain the interface data of the preview interface. Exemplarily, surface flinger can call the CPU or GPU to synthesize the layers. Surface flinger can also send the layers to be synthesized, such as the first layer and the third layer, to the hardware synthesis HWC, and the hardware synthesis HWC synthesizes the preview interface.

[0132] The surface flinger can place the rendered first interface in a buffer and set the buffer state including the first interface to ready. The display driver can periodically read the interface data included in the buffer with the ready state and trigger the display to display the interface.

[0133] S11, the display screen displays a preview interface.

[0134] The preview interface includes the second image. In this way, the mobile phone can display the preview interface including the HDR display effect in real time.

[0135] The present application embodiment provides an electronic device, which includes: a memory, a display screen, and one or more processors. The display screen is coupled to the processor. The memory is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various functions or steps executed by the mobile phone in the above method embodiment. The structure of the electronic device can refer to Figure 1 The structure of the electronic device 100 is shown.

[0136] The present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are in the above electronic device (such as Figure 1 When the method is executed on the electronic device 100 shown in the figure, the electronic device executes each function or step in the above method embodiment.

[0137] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step in the above method embodiment.

[0138] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instruction is executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.

[0139] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0144] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A shooting preview method, characterized in that: Applied to an electronic device, the electronic device includes a camera and a display screen, and a camera APP is installed in the electronic device, and the method includes: In response to the user opening the camera app, the following operations are performed: The camera captures a first image and sends the first image to a camera APP; the first image is a preview image for displaying on a preview interface of the camera APP; The camera APP generates a first LUT matching the first image; The camera APP calls the GPU to render the second layer of the first image off-screen; the camera APP uses the first LUT to process the first image in the second layer to obtain a second image; wherein the second dynamic imaging range of the second image is higher than the first dynamic imaging range of the first image; The camera APP calls the GPU to render the first layer of the second image, and synthesizes the first layer and the layer including the background image and interface elements of the camera APP to obtain and display a preview interface of the camera APP including the second image.

2. The method according to claim 1, characterized in that After acquiring the first image, the method further includes: Identifying whether a shooting scene of the first image is a high dynamic range shooting scene; The camera APP uses the first LUT to process the first image in the second layer to obtain a second image, including: If the shooting scene of the first image is a high dynamic imaging range shooting scene, the camera APP uses the first LUT to process the first image in the second layer to obtain the second image.

3. The method according to claim 1, characterized in that The camera APP uses the first LUT to process the first image in the second layer to obtain a second image, including: If the high dynamic range imaging HDR function of the electronic device is turned on, the camera APP uses the first LUT to process the first image in the second layer to obtain the second image.

4. The method according to any one of claims 1 to 3, characterized in that: The camera APP calls the GPU to render the first layer of the second image, including: The camera APP calls the GPU to render the first layer of the second image through a standard graphics library application programming interface OpenGL API.

5. The method according to any one of claims 1 to 3, characterized in that: The first LUT is used to map RGB values ​​of some or all pixels in the first image.

6. The method according to claim 1, characterized in that The camera APP generates a first LUT matching the first image, including: The camera APP obtains the first LUT based on the brightness information of the first image and the screen brightness value of the display screen; wherein the brightness information includes a histogram corresponding to the first image, or a brightness map corresponding to the first image.

7. The method according to any one of claims 1 to 3 and 6, characterized in that: The electronic device comprises a first renderer and a second renderer; The camera APP calls the GPU to render the second layer of the first image off-screen, including: the camera APP triggers the second renderer to call the GPU to render the second layer of the first image off-screen, and the electronic device does not display the rendered second layer; The camera APP uses the first LUT to process the first image in the second layer to obtain the second image, including: the camera APP triggers the first renderer to use the first LUT to process the first image into the second image; The camera APP calls the GPU to render the first layer of the second image, including: the camera APP triggers the first renderer to call the GPU to render the first layer of the second image.

8. The method according to claim 7, characterized in that The camera APP triggers the second renderer to call the GPU to render the second layer of the first image off-screen, including: The camera APP loads the embedded system image library EGL environment; The camera APP obtains the first image through the surface texture object; The camera APP establishes a correspondence between the first surface object of the first image and the first texture through the surface texture object; The camera APP triggers the second renderer to call the GPU to render the first texture off-screen based on the embedded system image library EGL statement through the standard graphics library application programming interface OpenGL API to obtain the second layer of the first image; wherein the second layer is cached in the first frame buffer object FBO; The camera APP triggers the first renderer to process the first image into a second image using a first LUT, including: The camera APP triggers the first renderer, calls the image enhancement algorithm, obtains the first LUT, and uses the first LUT to process the first image in the first frame buffer object FBO into the second image; The camera APP triggers the first renderer to call the GPU to render the first layer of the second image, including: The camera APP triggers the second renderer to call the GPU to render the second image in the first frame buffer object FBO through the standard graphics library application programming interface OpenGL API based on the embedded system image library EGL statement to obtain the first layer of the second image.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a display screen, a camera, and one or more processors; the camera, the memory, and the display screen are coupled to the processor; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images