Image processing method and related device

Abstract

The invention provides an image processing method and a related device, which are applied to electronic equipment with image processing capability, and a chip algorithm corresponding to a shooting mode is stored in a hardware abstraction layer of the electronic equipment, so that a third-party application program starts a shooting function of the electronic equipment, and after a target shooting mode is selected from multiple shooting modes, the shooting function of the electronic equipment is realized. Based on the support of an image information processor (ISP) in a hardware layer, a hardware abstraction layer can process image data reported by camera equipment based on an algorithm corresponding to the target shooting mode in a chip algorithm to obtain an image with a shooting effect, so that the image with the shooting effect is displayed in a preview interface of a third-party application program. It can be seen that according to the method, an underlying image processing chip (namely ISP) of the system camera is reused to achieve image algorithm processing, and compared with a software algorithm, the method has the higher processing speed and the better processing result.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method and related devices. Background Art

[0002] As the functions of electronic devices become increasingly sophisticated, electronic devices equipped with cameras (such as mobile phones) can not only open the camera in the camera application to shoot, but also call the camera through a third-party application to shoot, thereby collecting images. However, when the third-party application calls the camera to shoot, the shooting modes are limited, and there are generally problems such as poor shooting effects and poor shooting quality. Summary of the invention

[0003] An image processing method and related device provided in an embodiment of the present application can reuse the underlying image processing chip (i.e., ISP) of the system camera to implement image algorithm processing, with faster processing speed and better processing results.

[0004] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein a chip algorithm is stored in a hardware abstraction layer of the electronic device, and the method comprises:

[0005] In response to a user operation on a camera control in a first application, an image is captured by a camera of the electronic device, and a preview interface of the first application is displayed, wherein the first application is different from a camera application, a plurality of shooting modes are displayed in the preview interface, and the preview interface includes the image captured by the camera;

[0006] In response to a user operation on a target shooting mode, a first image captured by the camera is processed based on an algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image having a shooting effect of the target shooting mode, wherein the target shooting mode is one selected from the multiple shooting modes;

[0007] The second image is displayed in the preview interface.

[0008] It can be seen that the chip algorithm corresponding to the shooting mode is stored in the hardware abstraction layer. Therefore, when the first application starts the shooting function of the electronic device, the image data collected by the camera can be processed based on the algorithm corresponding to the target shooting mode to obtain a second image of the shooting effect of the target shooting mode. The underlying chip (that is, the dedicated chip) of the camera application of the reuse system is used to realize image post-processing, which can improve the processing speed and processing effect compared with the software algorithm (that is, the algorithm processed by the general chip).

[0009] In a possible implementation of the first aspect, after displaying the second image in the preview interface, the method further includes:

[0010] In response to a user operation on a shooting control in the preview interface, the second image is saved.

[0011] It can be seen that the chip algorithm based on the hardware abstraction layer storage saves the second image with the shooting effect for the first application.

[0012] In a possible implementation of the first aspect, the electronic device includes an application framework layer, the application framework layer includes a camera extension library, and the responding to a user operation on a camera control of a first application, capturing an image through a camera of the electronic device, and displaying a preview interface of the first application include:

[0013] In response to a user operation on a camera control of the first application, calling a camera interface to obtain a camera device list and properties of the camera devices in the camera device list;

[0014] Calling a camera extension interface to obtain camera capabilities supported by the hardware of the electronic device from the camera extension library;

[0015] Based on the camera capability and the properties of the camera device, an image is captured through the camera of the electronic device, and a preview interface of the first application is displayed.

[0016] In a possible implementation of the first aspect, based on the camera capability and the properties of the camera device, capturing an image through the camera of the electronic device and displaying a preview interface of the first application program includes:

[0017] Determining the camera capabilities supported by the camera device by determining, according to the attributes of the camera device, whether the camera device supports the camera capabilities supported by the hardware;

[0018] Based on the camera capabilities supported by the camera device, images are captured through the camera of the electronic device, and a preview interface of the first application is displayed, wherein each shooting mode displayed in the preview interface corresponds to the camera capabilities supported by the camera device.

[0019] It can be understood that the camera device includes a logical camera and a physical camera. The logical camera is an abstract device composed of multiple physical cameras (that is, the camera of the electronic device), and certain camera capabilities are completed by controlling multiple physical cameras at the same time. Therefore, the camera capabilities that can be completed by different logical cameras are different, and the attributes of the camera device can be used to describe the camera capabilities of the camera device. In addition, the camera capabilities supported by the hardware of the electronic device are stored in the camera extension library. Therefore, based on the camera capabilities and the attributes of the camera device, it can be determined whether each shooting mode is supported in the attributes of the camera device reported at the bottom layer, so that the shooting modes supported by the camera device and the images captured by the camera without algorithm processing can be displayed in the preview interface of the first application. Because the shooting modes displayed in the preview interface are obtained after judgment, the processing speed and response speed for the shooting mode can be improved.

[0020] In a possible implementation of the first aspect, in response to a user operation on a target shooting mode, processing a first image captured by the camera based on an algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image having a shooting effect of the target shooting mode includes:

[0021] In response to a user operation on a target shooting mode, powering on a target camera device, wherein the target camera device includes a camera among the cameras that supports a camera capability corresponding to the target shooting mode;

[0022] Acquire configuration parameters corresponding to the target shooting mode, and call the target camera device to collect image data based on the configuration parameters;

[0023] Based on the algorithm corresponding to the target shooting mode in the chip algorithm, the first image reported by the target camera device is processed to obtain a second image with the shooting effect of the target shooting mode.

[0024] It can be seen that based on the camera capabilities and the properties of the camera device, it can be determined whether each shooting mode is supported in the properties of the camera device reported by the bottom layer, thereby determining the camera capabilities supported by the camera device and displaying the corresponding shooting mode. Therefore, when the first application chooses to enter a certain shooting mode, the camera device corresponding to the shooting mode can be selected to directly open the camera, thereby improving the response speed.

[0025] In a possible implementation of the first aspect, the electronic device includes an image signal processor, and the algorithm corresponding to the target shooting mode in the chip algorithm is used to process the first image reported by the target camera device to obtain a second image having a shooting effect of the target shooting mode, including:

[0026] The hardware abstraction layer starts the sensor of the target camera device to collect an image light signal, and transmits the image light signal to an image signal processor;

[0027] The image signal processor processes the image light signal based on an algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image having a shooting effect of the target shooting mode.

[0028] It can be seen that the camera device corresponding to the target shooting mode is a device that supports the target shooting mode. Therefore, the hardware abstraction layer can directly call the corresponding sensor to collect image light signals. The collected image light signals meet the specifications corresponding to the target shooting mode, which can further improve the processing speed. Then the image signal processor processes the collected image light signals based on the algorithm corresponding to the target shooting mode stored in the chip algorithm. Because the image signal processor is a dedicated chip for image processing, the processing speed is fast and the processing effect is good, and the second image with the shooting effect can be quickly refreshed in the preview interface.

[0029] In a possible implementation of the first aspect, the electronic device includes an image signal processor, and the saving the second image in response to a user operation on a shooting control in the preview interface includes:

[0030] In response to a user operation on a shooting control in the preview interface, sending a shooting request to the hardware abstraction layer, wherein the shooting request contains shooting parameters corresponding to the target shooting mode, and the shooting parameters are obtained from the camera extension library;

[0031] The hardware abstraction layer starts the sensor of the target camera device based on the photographing parameter to collect an image light signal;

[0032] The image signal processor processes the image light signal based on an algorithm corresponding to the target shooting mode in the chip algorithm, and saves a second image having a shooting effect of the target shooting mode.

[0033] It can be seen that the camera device corresponding to the target shooting mode is a device that supports the target shooting mode. Therefore, the hardware abstraction layer can directly call the corresponding sensor to collect image light signals. The collected image light signals meet the specifications corresponding to the target shooting mode, which can further improve the processing speed. Then the image signal processor processes the collected image light signals based on the algorithm corresponding to the target shooting mode stored in the chip algorithm. Because the image signal processor is a dedicated chip for image processing, it can increase the speed of saving images.

[0034] In a possible implementation of the first aspect, the shooting mode includes one or more of a night scene mode, an HDR mode, a skin beautification mode, a blur mode, and an automatic mode.

[0035] In a second aspect, an electronic device is provided by an embodiment of the present application, the electronic device comprising: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method for drawing smooth rounded corners described in the first aspect or any possible implementation of the first aspect.

[0036] In a third aspect, the present application provides a chip or a chip system, which includes at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to execute the method for drawing smooth rounded corners described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin, or a circuit, etc.

[0037] In a possible implementation, the chip or chip system described above in the embodiment of the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0038] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the computer executes the method of drawing smooth rounded corners as described in the first aspect or any possible implementation of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a communication device, enables the communication device to execute a method for drawing smooth rounded corners as described in the first aspect or any possible implementation of the first aspect.

[0040] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments.

[0041] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The following is an introduction to the drawings used in the embodiments of the present application.

[0043] Figure 1 is a schematic diagram of a software architecture for implementing camera capabilities of a third-party application provided in an embodiment of the present application;

[0044] Figure 2 is a structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of the software architecture of the electronic device provided in the embodiment of the present application;

[0046] Figure 4A-4C A set of user interfaces for a third-party application to call a camera application provided in an embodiment of the present application;

[0047] Figure 5 is a timing diagram of an image processing method provided in an embodiment of the present application;

[0048] Figure 6 It is a flowchart of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0050] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0051] Usually, when third-party applications call the camera in the electronic device to take pictures, there are usually problems such as poor photo effects and limited shooting modes. The provided camera expansion interface implements a camera capability exposure solution on the camera of the electronic device, which is used to open some camera functions to third-party applications, thereby improving the photo effect and photo quality.

[0052] See also Figure 1 , Figure 1 Schematic diagram of a software architecture for implementing camera capabilities in a third-party application provided in an embodiment of the present application. Figure 1 It can be seen that Google The provided set of extended application programming interfaces (APIs) is used by original equipment manufacturers (OEMs) to provide capabilities downward, and the camera 2 extension API (Camera2Extension API) and camera X extension API (CameraX Extension API) are encapsulated to provide capability interfaces upward. Among them, camera capabilities include but are not limited to: Auto, skin beautification / face retouch, high-dynamic range (HDR), night scene / night (Night), blur / out-of-focus imaging, etc.

[0053] like Figure 1 As shown, the application layer of the electronic device includes a package of a third-party application that can call camera permissions, and the third-party application includes: a third-party camera 2 application and a third-party camera X application. In order to implement the camera capability, the third-party camera 2 application and / or the third-party camera X application first call the corresponding interface to obtain the preview stream (previewstream) and the capture stream (capture stream) collected by the camera device from the hardware abstraction layer (HAL). Exemplarily, for the third-party camera 2 application, the camera 2 API (Camera2 API) in the application framework layer can be called to obtain the preview stream and the capture stream from the HAL, wherein the preview stream and the capture stream include the image obtained after the photosensitive element in the camera converts the captured light source signal into an electrical signal, and the graphic can be called the original graphic, or a RAW image.

[0054] For example, for a third-party camera X application, the preview stream and the photo stream can be obtained from the HAL based on the Camera X API (CameraX API) by calling the Camera2 API. In order to realize the camera capability, after the Camera2 API or the CameraX API obtains the preview stream and the photo stream, the corresponding extension interface can be called to transmit the preview stream and the photo stream to the camera extension OEM library for camera algorithm processing, and finally the processed image data is returned to the third-party camera 2 application and / or the third-party camera X application, and finally the third-party camera 2 application and / or the third-party camera X application can display the processed image.

[0055] Exemplarily, the Camera2 API transmits the preview stream and the photo stream to the Camera2 extension API, the Camera2extension API calls the camera extension interface (camera extensions interface) to transmit the preview stream and the photo stream to the camera extension OEM library (camera extension OEM library) in the application framework layer for image algorithm processing, the camera extension OEM library transmits the processed image data to the Camera2 extension API, and finally returns it to the third-party Camera 2 application.

[0056] Exemplarily, the CameraX API transmits the preview stream and the photo stream to the CameraX extension API, and the CameraXextension API calls the camera extension interface (camera extensions interface) to transmit the preview stream and the photo stream to the camera extension OEM library in the application framework layer for image algorithm processing. The camera extension OEM library transmits the processed image data to the CameraX extension API, and finally returns it to the third-party camera 2 application.

[0057] It can be seen that based on Figure 1 In the software architecture shown, the images returned by the hardware abstraction layer are not processed by algorithms. The image post-processing algorithm of the camera extension OEM library in the application framework layer is needed to help third-party applications implement camera capabilities. This is pure software algorithm processing, which may result in slow processing speed and poor processing effect.

[0058] In view of this, the present application provides an image processing method, which is applied to an electronic device with image processing capabilities, and the hardware abstraction layer of the electronic device stores a chip algorithm corresponding to the shooting mode. Therefore, after a third-party application starts the shooting function of the electronic device and selects a target shooting mode from a plurality of shooting modes, based on the support of the image information processor (ISP) in the hardware layer, the hardware abstraction layer can process the image data reported by the camera device based on the algorithm corresponding to the target shooting mode in the chip algorithm to obtain an image with a shooting effect, thereby displaying the image with a shooting effect in the preview interface of the third-party application. It can be seen that the present application reuses the underlying image processing chip (i.e., ISP) of the system camera to realize the algorithm processing of the image, which has a faster processing speed and better processing results than the software algorithm.

[0059] First of all, the electronic device in the embodiments of the present application can be a smart screen device, a smart television (television, TV), a mobile phone, a tablet computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (personal digital assistant, PDA), a wearable device (such as: a smart watch, a smart bracelet), and other devices with display functions. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0060] For example, taking the electronic device as a mobile phone, Figure 2 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. That is, exemplary, Figure 2 The electronic device shown may be a mobile phone.

[0061] like Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 221, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0062] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0063] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0064] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0065] The processor 110 may also be provided with a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0066] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 221, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In another embodiment, the power management module 141 can also be set in the processor 110. In another embodiment, the power management module 141 and the charging management module 140 can also be set in the same device.

[0067] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0068] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In another embodiment, the antenna can be used in combination with a tuning switch.

[0069] The mobile communication module 150 can provide wireless communication solutions including second generation (2G) / third generation (3G) / fourth generation (4G) / fifth generation (5G) / sixth generation (6G) applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In one embodiment, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In one embodiment, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0070] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In one embodiment, the modem processor may be an independent device. In another embodiment, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0071] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0072] In one embodiment, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0073] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0074] The display screen 194 is used to display images, videos, user interfaces, controls, windows, etc. In one implementation, the display screen 194 is used to display a smooth rounded corner obtained by a smooth rounded corner drawing method provided in the present application. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In one embodiment, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0075] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0076] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In one embodiment, ISP can be set in camera 193.

[0077] In a possible implementation, the ISP may also select a corresponding image algorithm based on the selected shooting mode to perform algorithm optimization on the image to obtain a color matching effect corresponding to the shooting mode. The shooting mode includes one or more of a night scene mode, an HDR mode, a skin beautification mode, a blur mode, and an automatic mode.

[0078] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0079] The camera 193 may include one or more of the following cameras: a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, a zoom camera, or a depth camera. Among them, the telephoto camera has a small shooting range and is suitable for shooting distant objects; the wide-angle camera has a larger shooting range; the ultra-wide-angle camera has a larger shooting range than the wide-angle camera and is suitable for shooting panoramic and other large-screen scenes. The depth camera can be used to measure the object distance of the object to be photographed, that is, to measure the depth information of the object to be photographed, for example, it may include a three-dimensional (3D) depth camera, a time of flight (TOF) depth camera, or a binocular depth camera.

[0080] The camera 193 may include a main camera and a secondary camera. The main camera may be used to capture images, for example, it may include a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a zoom camera, etc. The secondary camera may be used for ranging or other auxiliary functions, for example, it may include a depth camera.

[0081] The camera 193 may include a front camera and / or a rear camera. The front camera may include one or more main cameras, and the rear camera may also include one or more main cameras. When capturing an image, the target main camera used by the electronic device to capture the image may be a default main camera or a main camera selected by the user.

[0082] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0083] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 221, and / or instructions stored in a memory provided in the processor.

[0084] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0085] The audio module 170 is used to convert digital audio information into analog audio signal output, and also used to convert analog audio input into digital audio signal. The audio module 170 can also be used to encode and decode audio signals.

[0086] The speaker 170A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0087] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals.

[0088] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0089] The earphone jack 170D is used to connect a wired earphone.

[0090] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In one embodiment, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In one embodiment, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0091] The gyro sensor 180B may be used to determine the motion posture of the electronic device 100. In one embodiment, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180B.

[0092] The air pressure sensor 180C is used to measure air pressure.

[0093] The magnetic sensor 180D includes a Hall sensor, and the electronic device 100 can detect the opening and closing of the flip leather case by using the magnetic sensor 180D.

[0094] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes).

[0095] The distance sensor 180F is used to measure the distance.

[0096] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100.

[0097] The ambient light sensor 180L is used to sense the brightness of ambient light.

[0098] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0099] The temperature sensor 180J is used to detect the temperature.

[0100] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0101] Bone conduction sensor 180M can obtain vibration signals.

[0102] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0103] The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0104] The software system of an electronic device (such as a mobile phone) can adopt a layered architecture, a transaction-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. This application embodiment takes the Android system with a layered architecture as an example to illustrate the software architecture of a mobile phone. Figure 3 , Figure 3 It is a schematic diagram of the software architecture of the electronic device provided in the embodiment of the present application.

[0105] like Figure 3 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer (application), the application framework layer (frame), the hardware abstraction layer (HAL), the driver layer, and the hardware layer. Among them:

[0106] The application layer (application) may include a series of program packages. For example, the application may include third-party applications, and may also include camera, gallery and other applications. Among them, the third-party application includes an application that can call the camera application to start the photo function, such as the third-party camera 2 application and the third-party camera X application. The camera application may include but is not limited to: UI module, photo module, gallery module, etc. The UI module may be a cameraUI module, which is mainly responsible for the human-computer interaction of the camera application, such as controlling the display of the preview interface and the preview screen therein, receiving and responding to user operations occurring in the preview interface. The photo module provides photo function, focus function, etc. The gallery module can be used to store the photos taken by the user in the file system or a specific database of the electronic device for retrieval by applications such as the gallery.

[0107] Among them, the third-party Camera 2 application uses the Camera2 API to call the camera function, and the third-party Camera X application uses the Camera X interface (also known as the CameraX API) or the Camera X extension interface (also known as the CameraXextension API) to call the camera function. The CameraX API is integrated in the application layer.

[0108] The application framework layer (frame) provides application programming interfaces (API) and programming frameworks for applications in the application layer. It mainly involves the camera framework, which may include camera access interfaces such as camera extension libraries and camera services. It plays a role of connecting the upper and lower parts. It can interact with camera applications through the application API, and can also interact with HAL through the HAL interface definition language (HIDL).

[0109] The camera access interface includes the Camera 2 API, the Camera 2 extension API and the Camera extension API.

[0110] The camera extension library is an implementation library of the open standard interface of camera capabilities, providing third-party applications with camera functions such as automatic, skin beautification, blur, HDR, and night scene. The camera extension library is used to determine whether the camera capabilities corresponding to the requests from third-party applications are supported, report the distribution stream resolution, report the distribution stream, and send the parameters required for taking pictures.

[0111] The Camera Service is used to respond to requests from the application layer and provide session management, device management, and provider management. The Camera Service is an independent process that, as a server, handles cross-process requests from application clients and performs certain operations internally, and then sends the request again to the HAL process as a server as a client. Therefore, the Camera Service belongs to the channel part of the camera and plays a connecting role.

[0112] The Hardware Abstraction Layer (HAL) is an interface layer between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. For example, the Hardware Abstraction Layer receives requests from the Camera Provider, sends the requests to the driver, and waits for the results to be returned before reporting them to the Camera Provider.

[0113] Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer and a chip algorithm library. Among them, the camera hardware abstraction layer may provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera) and more camera devices. The camera algorithm library stores a variety of image processing algorithms. For example, in an embodiment of the present application, the camera algorithm library may include an automatic algorithm, a skin beautification algorithm, a blur algorithm, a night scene algorithm and an HDR algorithm. Among them:

[0114] The automatic algorithm is used to implement the automatic camera capability. When a third-party application calls the camera application to shoot, the final image can be automatically adjusted according to the surrounding scenery.

[0115] The skin beautification algorithm is used to implement the skin beautification camera capability. When a third-party application calls the camera application to take a still image, it modifies the facial color, contour, etc.

[0116] The blur algorithm is used to implement the blur feature of the camera, making foreground characters clearer when a third-party application calls the camera app to take photos.

[0117] The night scene algorithm is used to implement the night scene camera capability, which allows third-party applications to call the camera application to capture the best static images in low-light environments (such as at night).

[0118] The HDR algorithm is used to implement the HDR camera capability, and different automatic exposure (AE) configurations are used when third-party applications call the camera to take pictures to obtain the best effect.

[0119] The driver layer is a layer between hardware and software, which includes drivers for various hardware. The driver layer may include camera drivers, digital signal processor drivers, and image processor drivers. Among them, the camera driver is used to drive the image sensor of one or more cameras in the camera to collect images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0120] The hardware layer may include a camera module, an image signal processor, a digital signal processor, and an image processor. The camera may include one or more image sensors of the camera (e.g., image sensor 1, image sensor 2, etc.). Optionally, the camera may also include a camera motor, a lens, a TOF sensor, and the like.

[0121] The following is an example of the workflow of electronic device software and hardware, using a scenario where a third-party application calls a camera application.

[0122] In the embodiment of the present application, when the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer, and the kernel layer processes the user operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information), and identifies the control corresponding to the input event. Taking the touch operation as a user operation of a camera control in a third-party application as an example, the third-party application calls the camera extension interface of the application framework layer, starts the camera capabilities provided in the camera extension library, and then sends an instruction to start the camera capability by calling the camera device in the camera hardware abstraction layer, such as camera device 1. The camera hardware abstraction layer sends the instruction to the camera driver of the driver layer, and the camera driver can start the sensor corresponding to the camera device, such as sensor 1, and then collect the image light signal through the sensor, and transmit the image light signal to the image signal processor for preprocessing to obtain a RAW image, and then transmit the original stream back to the hardware abstraction layer through the camera driver.

[0123] The hardware abstraction layer can send the RAW image stream to the chip algorithm library. Based on the support of the image signal processor, the chip algorithm library in the hardware abstraction layer can perform algorithm optimization processing on the RAW image stream, such as beauty / body beautification algorithm, automatic algorithm, night scene algorithm, HDR algorithm, blur algorithm, etc., to obtain better images. The hardware abstraction layer returns the above image data to the third-party application through the camera extension interface. Subsequently, the third-party application and the gallery application can display photos with camera capabilities to the user, such as beautified photos, etc., with the support of the window manager.

[0124] It can be seen that Figure 3 The software architecture shown is for Figure 1 For the software architecture shown in the figure, the electronic device can reuse the image signal processor of the system camera in the hardware abstraction layer to perform algorithmic processing on the image sent back by the camera device. Therefore, the image returned to the camera service by the hardware abstraction layer has been processed by the algorithm, and does not need to be processed in the application framework layer, and can be directly returned to the third-party application through the camera service.

[0125] Figure 4A-4C It is a user interface for a group of third-party applications to call a camera application provided by an embodiment of the present application.

[0126] Figure 4A It is a homepage of an electronic device provided in an embodiment of the present application. Figure 4A As shown, the main interface 113 may include a status bar 111 , a page indicator 112 , and a plurality of application icons.

[0127] Among them, the status bar 111 may include one or more signal strength indicators of a mobile communication signal (also referred to as a cellular signal), a wireless fidelity (Wi-Fi) signal strength indicator, a battery status indicator, a time indicator, etc.

[0128] The page indicator 112 may be used to indicate the positional relationship between the currently displayed page and other pages.

[0129] Multiple application icons may include a settings application icon 113A, an application market application icon 113B, a browser application icon 113C, a third-party application icon 113D, a gallery application icon 113E, a camera application icon 113F, an address book application icon, a phone application icon, a message application icon, etc. Not limited to the above icons, the main interface 113 may also include other application icons, which are not listed here one by one. Multiple application icons may be distributed on multiple pages. The page indicator 112 can be used to indicate which page the user is currently browsing is one of the multiple pages carrying multiple applications. The user can browse other pages by sliding left and right. Among them, the third-party application is an application with a photo-taking function, that is, the third-party application includes a camera control that can call a camera application.

[0130] Understandably, Figure 4A The user interface described below merely illustrates a possible user interface style of an electronic device taking a mobile phone as an example and should not constitute a limitation on the embodiments of the present application.

[0131] like Figure 4A As shown, the electronic device may detect a user operation for acting on the third-party application icon 113D, and in response to the above operation, the electronic device may display a Figure 4B The user interface shown. Figure 4B It is the main interface of a third-party application provided in the embodiment of the present application. Figure 4B As shown, the main interface 114 may include a camera control 1141, which is used to call the camera application of the electronic device to take a photo in the third application. The electronic device may detect the user operation acting on the camera control 1141, and in response to the above operation, the electronic device may display a Figure 4C The user interface shown. Figure 4C It is a user interface provided by an embodiment of the present application for a third-party application to call the shooting and display services provided by the camera application, also known as a preview interface.

[0132] like Figure 4C As shown, the preview interface may include a mode bar 121 , a shooting control 122 , a preview window 123 , and a flip control 124 .

[0133] The mode bar 121 may include multiple shooting mode options, which are used to indicate that the electronic device supports the camera capabilities provided to third-party applications. For example, the shooting mode options include but are not limited to: automatic mode option 1211, night scene mode option 1212, skin beauty mode option 1213, HDR mode option 1214, blur mode option 1215, and the like. Among them, different shooting modes can provide users with shooting services with different effects, and users can select any shooting mode from multiple shooting modes in third-party applications according to different needs to shoot. It is understandable that the above one or more shooting mode selections can be expressed as text information on the interface, such as: "automatic", "night scene", "skin beauty", "HDR", "blur", and can also be expressed as images or other forms of interactive elements, which are not limited in this application.

[0134] The electronic device can detect the user operation acting on the mode bar 121 and change the currently used shooting mode according to the above user operation. The above user operation is, for example, a left swipe / right swipe operation. For example, when it is detected that the drag mode bar 121 is slid to the left (left swipe operation) and the float 125 stops at the "blur" option, the electronic device can switch to the "blur" mode. By default, the electronic device 100 first uses the normal mode without any shooting effect.

[0135] The shooting control 122 is used to trigger taking a photo. The electronic device can detect whether there is a user operation acting on the shooting control 122, such as a click operation. When the user operation acting on the shooting control 122 is detected, the electronic device 100 can generate a shooting instruction. The electronic device 100 can obtain the image reported by the camera corresponding to the timestamp according to the shooting instruction, and then save it as a photo. In summary, the user operation acting on the shooting control 122 is the shooting operation described in this application.

[0136] The preview window 123 can be used to display the image reported by the camera in real time. In different shooting modes, the electronic device 100 can process the image reported by the camera based on the chip algorithm stored in the hardware abstraction layer to improve the display effect of the image. Figure 4C It can be seen that in the "blur" mode, the electronic device 100 can blur the background in the image reported by the camera to highlight the foreground person, and the preview window 123 of the third-party application can display an image with a "blur" effect ( Figure 4C (The “blurring” effect is not shown). It is understandable that, before the shooting mode is selected, the preview window 123 displays the image reported by the default camera, that is, the image that has not been processed by the algorithm optimization.

[0137] The flip control 124 can be used to monitor the user operation that triggers the flip camera. The electronic device can detect the user operation acting on the flip control 124, such as a touch operation. In response to the operation, the electronic device can switch the camera used for shooting, such as switching the rear camera to the front camera, or switching the front camera to the rear camera. Among them, different cameras support different shooting modes, that is, the front camera or the rear camera can support one or more of the above shooting modes. The specific support capability can be determined by referring to the following Figure 5 , I will not go into details here.

[0138] It is understandable that the preview window 123 can display the image processed by the image processing algorithm corresponding to different shooting modes in real time, so that the user can perceive the shooting effects corresponding to different shooting modes in real time, thereby improving the shooting effect and shooting quality of the third-party application.

[0139] For example, when a user operation acting on the automatic mode option 1211 is detected, the electronic device can switch to the "automatic" mode, process the image collected by the camera based on the image processing algorithm corresponding to the automatic mode in the hardware abstraction layer, automatically adjust the final image according to the surrounding scenery, and display the image processed by the image processing algorithm corresponding to the automatic mode in real time in the preview window 123, so that the user can feel the shooting effect corresponding to the automatic mode in real time. When a user operation acting on the shooting control 122 is detected, the electronic device can generate a shooting instruction, and the electronic device can obtain the image corresponding to the automatic mode according to the shooting instruction, and then save it as a photo.

[0140] When a user operation acting on the night scene mode option 1212 is detected, the electronic device can switch to the "night scene" mode, process the image collected by the camera based on the image processing algorithm corresponding to the night scene mode in the hardware abstraction layer, thereby adjusting the light brightness to adapt to the low-light environment, and display the image processed by the image processing algorithm corresponding to the night scene mode in real time in the preview window 123, so that the user can feel the shooting effect corresponding to the night scene mode in real time. When a user operation acting on the shooting control 122 is detected, the electronic device can generate a shooting instruction, and the electronic device can obtain the image corresponding to the night scene mode according to the shooting instruction, and then save it as a photo.

[0141] When a user operation acting on the skin beautification mode option 1213 is detected, the electronic device can switch to the "skin beautification" mode, and process the image collected by the camera based on the image processing algorithm corresponding to the skin beautification mode in the hardware abstraction layer, so as to adjust the facial color, contour, etc. of the person, and display the image processed by the image processing algorithm corresponding to the skin beautification mode in real time in the preview window 123, so that the user can feel the shooting effect corresponding to the skin beautification mode in real time. When a user operation acting on the shooting control 122 is detected, the electronic device can generate a shooting instruction, and the electronic device can obtain the image corresponding to the skin beautification mode according to the shooting instruction, and then save it as a photo.

[0142] When a user operation acting on the HDR mode option 1214 is detected, the electronic device can switch to the "HDR" mode, process the image collected by the camera based on the image processing algorithm corresponding to the HDR mode in the hardware abstraction layer, so as to use different automatic exposure configurations to obtain the best effect, and display the image processed by the image processing algorithm corresponding to the HDR mode in real time in the preview window 123, so that the user can feel the shooting effect corresponding to the HDR mode in real time. When a user operation acting on the shooting control 122 is detected, the electronic device can generate a shooting instruction, and the electronic device can obtain the image corresponding to the HDR mode according to the shooting instruction, and then save it as a photo.

[0143] Based on the software and hardware architecture of the above electronic device, an image processing method provided by this application is introduced.

[0144] Figure 5 is a timing diagram of an image processing method provided in an embodiment of the present application. The implementation of the method can be based on the interactive cooperation between the application layer (such as a third-party application), the application framework layer (framework), and the hardware abstract layer (HAL) in the electronic device, such as Figure 5 As shown, the method may include:

[0145] Third-party applications call the camera

[0146] S1, third-party applications start the camera function.

[0147] Specifically, the system desktop can detect the user's operation of opening a third-party application, such as the user clicking on the desktop image of the "third-party application" to display the main interface of the third-party application. The main interface of the third-party application includes a camera control for calling the camera function. The electronic device detects the operation of starting the camera control, such as the user clicking on the interface element of the "camera control", and displays the preview interface of the third-party application. The preview interface can be as follows: Figure 4C The display includes a mode bar 121, a shooting control 122, a preview window 123, and the like.

[0148] S2, the third-party application requests the camera service for a list of camera devices.

[0149] Specifically, when a third-party application starts the camera function, the third-party application can call the camera interface (such as Camera2 API or CameraX API) to obtain a list of available camera devices (Camera IDs), which stores all camera IDs owned by the electronic device. The camera ID includes a physical camera ID and a logical camera ID. A logical camera is an abstract device composed of multiple physical cameras. A logical camera completes certain camera capabilities by controlling multiple physical camera devices at the same time. The logical camera ID is a unique string that represents the abstract capabilities of multiple physical cameras.

[0150] S3: The camera service returns a list of camera devices to the third-party application.

[0151] Specifically, the camera service calls the camera interface to return a camera device list to the third-party application.

[0152] S4, the third-party application requests the camera service for camera attributes supported by the camera device.

[0153] Specifically, third-party applications can call the camera interface for each Camera ID to obtain the camera attribute set it supports (get Camera Characteristics). The attributes of each camera device are composed of key-value pairs. The key is also called an attribute tag, which is used to obtain the corresponding attribute value. The value is used to describe the specific value of the capability value. The value can be an integer or an array collection. Among them, Camera Characteristics is a property class that describes the camera device, and the attributes in it are fixed.

[0154] S5: The camera service returns the camera properties supported by the camera device to the third-party application.

[0155] Specifically, the camera service calls the camera interface to return the camera properties supported by the camera device to the third-party application.

[0156] S6, the third-party application requests the camera capabilities supported by the underlying layer from the camera extension library.

[0157] Specifically, the third application can call the camera extension interface (such as Camera2 extension API or CameraX extension API) to obtain the camera capabilities (get supported extensions) supported by the underlying layer (that is, the hardware of the electronic device) from the camera extension library. The camera extension interface determines whether each camera capability is supported in the reported Camera Characteristics by calling the (is Extension Available) interface. Finally, the camera capabilities supported by each Camera ID can be determined from the camera capability set (CameraCharacteristics) corresponding to each Camera ID through the (isExtension Available) interface. Among them, camera capabilities include but are not limited to: automatic, skin beautification, blur (portrait), HDR and night scene, etc.

[0158] Generally speaking, only one Camera ID on the rear camera supports all camera capabilities.

[0159] In one implementation, a third-party application calls the (is Extension Available) interface to determine whether a Camera ID supports a certain camera capability and thus supports a certain shooting mode through two capability tags. The parameters passed by the third-party application to the camera extension library include the Camera ID, the camera attribute set supported by the Camera ID, and the camera capabilities supported by the underlying layer. Among them:

[0160] Exemplarily, the first tag can be the camera capability supported by the underlying layer, such as com.hihonor.device.capabilities.extensionType. The value corresponding to the first tag can be an array set, such as [0,1,2,3,4], where 0 represents automatic in the camera capability, 1 represents skin beautification in the camera capability, 2 represents blur / portrait in the camera capability, 3 represents HDR in the camera capability, and 4 represents night scene in the camera capability. Therefore, the array set [0,1,2,3,4] can indicate that the Camera ID supports camera capabilities such as automatic, skin beautification, blur (portrait), HDR and night scene.

[0161] Exemplarily, the first tag can be a property of the camera device, such as com.hihonor.device.capabilities.cameraIdCustomInfo. The value corresponding to the second tag can also be an array set, which uses a series of rules to represent the scene modes (scene Mode) supported by the Camera ID. For example, scene Mode 0 indicates the normal shooting mode channel, and the corresponding camera capabilities (i.e., extension Type) are automatic, HDR, and night scene; sceneMode 23 indicates the portrait shooting mode channel, and the corresponding camera capabilities (i.e., extension Type) are skin beauty and portrait.

[0162] Therefore, if the first tag indicates that the Camera ID supports a certain camera capability, and the second tag indicates that the Camera ID supports the scene mode corresponding to the camera capability, then the Camera ID supports the camera capability. For example, the rear camera supports scene Mode 0 with Camera ID 5, and Camera ID5 has camera capabilities such as automatic, HDR, and night scene. Therefore, when a third-party application enters one of the modes such as automatic, HDR, and night scene, it will select Camera ID 5 to open the camera.

[0163] S7, the camera extension library returns the camera capabilities supported by the camera device to third-party applications.

[0164] Specifically, the camera extension library calls the camera extension interface to return the camera capabilities supported by the camera device to the third-party application.

[0165] Enter shooting mode with third-party apps

[0166] S8, third-party apps determine shooting mode.

[0167] Specifically, the preview interface of the third-party application may display multiple shooting mode options supported by the camera device, such as: automatic, skin beautification, blur (portrait), HDR and night scene, etc. The third-party application detects a user operation acting on a certain shooting mode and may change the currently used shooting mode according to the user operation.

[0168] When a third-party application enters an extended shooting mode, it will go through the process of opening the camera (opencamera), streaming, and previewing. Opening the camera means opening the Camera ID device corresponding to the shooting mode, and powering on the corresponding device at the bottom layer, such as powering on the camera sensor of the device.

[0169] Stream allocation generally involves configuring preview stream and photo stream. It can be understood as creating two channels for preview and photo, respectively, for sending requests and returning data.

[0170] S9, the third-party application requests the configuration parameters corresponding to the shooting mode from the camera extension library.

[0171] Specifically, before the third application sends a message to the hardware abstraction layer to start streaming, the third-party application can call the camera extension interface to obtain the configuration parameters corresponding to the selected shooting mode from the camera extension library. The configuration parameters include the resolution of the images transmitted in the preview stream and the photo stream, the parameters required for streaming, and so on.

[0172] Exemplarily, the camera extension interface obtains the allocated stream resolution from the camera extension library through the getSupportedResolutions method, that is, obtains the resolution supported by the bottom layer from the camera property set (Camera Characteristics) reported by the bottom layer, and then selects the correct resolution through a certain algorithm.

[0173] Exemplarily, the camera extension interface obtains the streaming parameters from the camera extension library through the onPresetSession method, wherein the streaming parameters include private parameters and public parameters corresponding to the shooting mode.

[0174] S10. The camera extension library returns configuration parameters corresponding to the shooting mode to the third-party application.

[0175] The configuration parameters corresponding to the shooting mode include the streaming resolution and the streaming parameters, wherein the streaming parameters include the private parameters and public parameters corresponding to the shooting mode. The camera extension library calls the camera extension interface to return the configuration parameters corresponding to the shooting mode to the third-party application.

[0176] S11, the third-party application sends a preview request corresponding to the shooting mode to the hardware abstraction layer.

[0177] The preview request carries the stream resolution and stream parameters corresponding to the preview stream, where the stream parameters include private parameters and public parameters corresponding to the shooting mode. The third-party application calls the camera interface to send a preview request corresponding to the shooting mode to the camera service, and the camera service sends a preview request corresponding to the shooting mode to the hardware abstraction layer.

[0178] Exemplarily, the third-party application passes a message to start streaming to the hardware abstraction layer by calling the createCaptureSession interface. Streaming may include the hardware abstraction layer configuring the preview stream using the streaming resolution and streaming parameters.

[0179] S12, the hardware abstraction layer processes the reported image based on the chip algorithm library to obtain a shooting effect corresponding to the shooting mode.

[0180] Specifically, after the hardware abstraction layer receives the message that triggers the start of the distribution stream, it can send an instruction to the camera device to start the camera device to start collecting images. After the camera device starts collecting images, the hardware abstraction layer can receive the image stream from the camera device, and the image stream at this time can be called the original image stream. The hardware abstraction layer can process and compress the original image stream from the camera device to obtain a preview stream. In addition, the preview stream can be configured using the distribution stream resolution and distribution stream parameters passed by the third-party application. Among them, the distribution stream parameters include private parameters and public parameters corresponding to the shooting mode, and the chip algorithm library stores image algorithms corresponding to multiple shooting modes. The hardware abstraction layer can determine the image algorithm corresponding to the shooting mode based on the private parameters corresponding to the shooting mode, and with the support of the image information processor, perform algorithmic processing on the image in the preview stream to obtain the shooting effect corresponding to the shooting mode.

[0181] S13, the hardware abstraction layer returns the image with the shooting effect to the third-party application.

[0182] Specifically, the hardware abstraction layer returns the image with the shooting effect to the camera service, and the camera service calls the camera interface to return the image with the shooting effect to the third-party application.

[0183] S14, a third-party application displays an image with shooting effects in the preview window.

[0184] Among them, the preview image in the preview window has the shooting effect corresponding to the shooting mode and can be continuously refreshed with the preview stream.

[0185] It is understandable that the essence of preview is to continuously send request requests to the hardware abstraction layer, and then the hardware abstraction layer continuously returns the image process, while taking a photo only sends a request request.

[0186] S15, the third-party application requests the camera extension library for shooting parameters.

[0187] Specifically, the third-party application can detect the user's photo operation, such as the user pressing the "photo" control in the preview interface of the third-party application, thereby triggering the photo. Before sending a photo request, the third-party application will first call the camera extension interface to obtain the photo parameters required to send the photo request from the camera extension library. Among them, the photo parameters include the stream resolution and stream parameters corresponding to the photo stream, and the stream parameters include private parameters and public parameters corresponding to the shooting mode. For example, the third-party application calls the getCaptureStages interface to obtain the photo parameters required to send the photo request from the camera extension library.

[0188] S16, the camera extension library returns the photo taking parameters to the third-party application.

[0189] Specifically, the camera extension library calls the camera extension interface to return the photographing parameters to the third-party application.

[0190] S17, the third-party application sends a photo request corresponding to the shooting mode to the hardware abstraction layer.

[0191] The photo request carries the corresponding stream resolution and stream parameters of the photo stream, as well as the photo timestamp. The stream parameters include private parameters and public parameters corresponding to the shooting mode. The third-party application calls the camera interface to send the photo request corresponding to the shooting mode to the camera service, and the camera service sends the photo request corresponding to the shooting mode to the hardware abstraction layer.

[0192] Regarding the description of the getCaptureStages interface, this interface can encapsulate the private parameters required for taking pictures and report them to third-party applications. The third-party application will use the private parameters and public parameters as the parameters of the photo request and send them to the hardware abstraction layer, so that the hardware abstraction layer knows what kind of processing needs to be performed on this frame of image. For third-party applications that do not have extended camera capabilities, it can only obtain public parameters to send photo requests, so it cannot use various underlying algorithms to process images, and there will be no shooting effects corresponding to various shooting modes. The following are some examples of parameters:

[0193]

[0194] S18, the hardware abstraction layer processes the image corresponding to the photo request based on the chip algorithm library to obtain a shooting effect corresponding to the shooting mode.

[0195] Specifically, the hardware abstraction layer can receive an image stream from a camera device, and the image stream can be called a raw image stream. The hardware abstraction layer can process and compress the raw image stream from the camera device to obtain an image stream. In addition, the image stream corresponding to the photo timestamp can be configured using the stream resolution and stream parameters passed by a third-party application. Among them, the stream parameters include private parameters and public parameters corresponding to the shooting mode, and the chip algorithm library stores image algorithms corresponding to a variety of shooting modes. The hardware abstraction layer can determine the image algorithm corresponding to the shooting mode based on the private parameters corresponding to the shooting mode, and with the support of the image information processor, perform algorithmic processing on the image stream corresponding to the timestamp to obtain the shooting effect corresponding to the shooting mode, and generate a thumbnail of the image.

[0196] S19, the hardware abstraction layer returns an image with shooting effects corresponding to the photo request to the third-party application.

[0197] Specifically, the hardware abstraction layer returns an image with shooting effects and a thumbnail of the image to the camera service, and the camera service calls a camera interface to return an image with shooting effects and a thumbnail of the image to the third-party application.

[0198] S20, third-party apps save images with camera effects.

[0199] See also Figure 6 , Figure 6 is a flowchart of an image processing method provided by an embodiment of the present application, which is applied to Figure 2 The hardware architecture shown and Figure 3 The software architecture shown includes but is not limited to the following steps:

[0200] Step S601: In response to a user operation on a camera control in a first application, an image is captured through a camera of the electronic device, and a preview interface of the first application is displayed.

[0201] The first application may be a third-party application with a shooting function, and the camera control is used to call the camera application of the electronic device, thereby capturing images through the camera of the electronic device and displaying a preview interface of the first application. The preview interface displays multiple shooting modes and also displays images captured by the camera.

[0202] Exemplarily, the shooting mode includes one or more of a night scene mode, an HDR mode, a skin beautification mode, a blur mode, and an automatic mode. Figure 4C The shooting mode is displayed in the mode column 121 shown.

[0203] In a possible implementation, the electronic device includes an application framework layer, and the application framework layer includes a camera extension library. For detailed description, please refer to Figure 3 In response to the user operation of the camera control of the first application, the camera interface is called to obtain the camera device list and the properties of the camera devices in the camera device list; the camera extension interface is called to obtain the camera capabilities supported by the hardware of the electronic device from the camera extension library; based on the camera capabilities and the properties of the camera device, an image is captured by the camera of the electronic device, and a preview interface of the first application is displayed. Among them, the image captured by the camera displayed in the preview interface at this time can be an image captured by the default camera without algorithm processing.

[0204] In one implementation, the camera capabilities supported by the camera device are determined by judging whether the camera device supports the camera capabilities supported by the hardware based on the attributes of the camera device; based on the camera capabilities supported by the camera device, images are captured through the camera of the electronic device, and a preview interface of a third-party application is displayed, wherein each shooting mode displayed in the preview interface corresponds to the camera capabilities supported by the camera device.

[0205] For detailed description of step S601, please refer to Figure 5 S1-S7 shown.

[0206] Step S602: In response to a user operation on a target shooting mode, a first image captured by the camera is processed based on an algorithm corresponding to the target shooting mode in a chip algorithm to obtain a second image having a shooting effect of the target shooting mode.

[0207] In a possible implementation, in response to a user operation on a target shooting mode, a target camera device is powered on, wherein the target camera device includes a camera that supports the camera capability corresponding to the target shooting mode; configuration parameters corresponding to the target shooting mode are obtained, and the target camera device is called based on the configuration parameters to collect image data; based on an algorithm corresponding to the target shooting mode in a chip algorithm, a first image reported by the target camera device is processed to obtain a second image having the shooting effect of the target shooting mode. The target shooting mode is one selected from a plurality of shooting modes, that is, the shooting mode in which the user operation is detected.

[0208] In one implementation, the electronic device includes an image signal processor, and the hardware abstraction layer starts the sensor of the target camera device to collect the image light signal and transmits the image light signal to the image signal processor; the image signal processor processes the image light signal based on the algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image with the shooting effect of the target shooting mode.

[0209] For detailed description of step S602, please refer to Figure 5 S8-S12 shown.

[0210] Step S603: Display the second image in the preview interface.

[0211] For detailed description of step S603, please refer to Figure 5 S13-S14 shown.

[0212] After step S603, the method further includes: saving the second image in response to a user operation on a shooting control in the preview interface.

[0213] Exemplarily, in response to a user operation on a shooting control in a preview interface, a shooting request is sent to the hardware abstraction layer, wherein the shooting request contains shooting parameters corresponding to a target shooting mode, and the shooting parameters are obtained from a camera extension library; the hardware abstraction layer starts the sensor of the target camera device based on the shooting parameters to collect image light signals; the image signal processor processes the image light signals based on an algorithm corresponding to the target shooting mode in the chip algorithm, and saves a second image having the shooting effect of the target shooting mode. For a detailed description, please refer to Figure 5 S15-S20 shown.

[0214] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0215] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0216] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0217] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0218] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0219] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0220] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0221] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0222] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0223] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0224] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0225] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0226] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0227] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0228] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0229] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that: Applied to an electronic device, a chip algorithm is stored in a hardware abstraction layer of the electronic device, and the method comprises: In response to a user operation on a camera control in a first application, an image is captured by a camera of the electronic device, and a preview interface of the first application is displayed, wherein the first application is different from a camera application, a plurality of shooting modes are displayed in the preview interface, and the preview interface includes the image captured by the camera; In response to a user operation on a target shooting mode, a first image captured by the camera is processed based on an algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image having a shooting effect of the target shooting mode, wherein the target shooting mode is one selected from the multiple shooting modes; The second image is displayed in the preview interface.

2. The method according to claim 1, characterized in that After displaying the second image in the preview interface, the method further includes: In response to a user operation on a shooting control in the preview interface, the second image is saved.

3. The method according to claim 1 or 2, characterized in that: The electronic device includes an application framework layer, the application framework layer includes a camera extension library, and the method of responding to a user operation on a camera control of a first application, capturing an image through a camera of the electronic device, and displaying a preview interface of the first application includes: In response to a user operation on a camera control of the first application, calling a camera interface to obtain a camera device list and properties of the camera devices in the camera device list; Calling a camera extension interface to obtain camera capabilities supported by the hardware of the electronic device from the camera extension library; Based on the camera capability and the properties of the camera device, an image is captured through the camera of the electronic device, and a preview interface of the first application is displayed.

4. The method according to claim 3, characterized in that The step of acquiring an image through the camera of the electronic device based on the camera capability and the property of the camera device, and displaying a preview interface of the first application program includes: Determining the camera capabilities supported by the camera device by determining, according to the attributes of the camera device, whether the camera device supports the camera capabilities supported by the hardware; Based on the camera capabilities supported by the camera device, images are captured through the camera of the electronic device, and a preview interface of the first application is displayed, wherein each shooting mode displayed in the preview interface corresponds to the camera capabilities supported by the camera device.

5. The method according to any one of claims 1 to 4, characterized in that: The method of processing the first image captured by the camera based on the algorithm corresponding to the target shooting mode in the chip algorithm in response to the user operation on the target shooting mode to obtain a second image having the shooting effect of the target shooting mode includes: In response to a user operation on a target shooting mode, powering on a target camera device, wherein the target camera device includes a camera among the cameras that supports a camera capability corresponding to the target shooting mode; Acquire configuration parameters corresponding to the target shooting mode, and call the target camera device to collect image data based on the configuration parameters; Based on the algorithm corresponding to the target shooting mode in the chip algorithm, the first image reported by the target camera device is processed to obtain a second image with the shooting effect of the target shooting mode.

6. The method according to claim 5, characterized in that The electronic device includes an image signal processor, and the algorithm corresponding to the target shooting mode in the chip algorithm is used to process the first image reported by the target camera device to obtain a second image having the shooting effect of the target shooting mode, including: The hardware abstraction layer starts the sensor of the target camera device to collect an image light signal, and transmits the image light signal to an image signal processor; The image signal processor processes the image light signal based on an algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image having a shooting effect of the target shooting mode.

7. The method according to claim 5, characterized in that The electronic device includes an image signal processor, and the step of saving the second image in response to a user operation on a shooting control in the preview interface includes: In response to a user operation on a shooting control in the preview interface, sending a shooting request to the hardware abstraction layer, wherein the shooting request contains shooting parameters corresponding to the target shooting mode, and the shooting parameters are obtained from the camera extension library; The hardware abstraction layer starts the sensor of the target camera device based on the photographing parameter to collect an image light signal; The image signal processor processes the image light signal based on an algorithm corresponding to the target shooting mode in the chip algorithm, and saves a second image having a shooting effect of the target shooting mode.

8. The method according to any one of claims 1 to 7, characterized in that: The shooting modes include one or more of a night scene mode, an HDR mode, a skin beautification mode, a blur mode and an automatic mode.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1-8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-8.

11. A computer program product comprising instructions, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 8.

12. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 8.

Images