An image processing method and related apparatus

Abstract

The application provides an image processing method and related device, which is applied to an electronic device with image processing capability. A chip algorithm corresponding to a shooting mode is stored in a hardware abstraction layer of the electronic device. Therefore, after a third-party application program starts a shooting function of the electronic device and selects a target shooting mode from multiple shooting modes, based on the support of an image signal processor (ISP) in the hardware layer, the hardware abstraction layer can process image data reported by a camera device based on an algorithm corresponding to the target shooting mode in the chip algorithm, to obtain an image with a shooting effect, so that the image with the shooting effect is displayed in a preview interface of the third-party application program. It can be seen that the application reuses a bottom image processing chip (i.e. the ISP) of a system camera to implement algorithm processing of an image, and has faster processing speed and better processing result compared with a software algorithm.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] As electronic devices become increasingly sophisticated, those equipped with cameras (such as mobile phones) can not only use camera apps to take pictures, but also access the camera through third-party applications. However, third-party applications often suffer from limited shooting modes and poor image quality when using the camera. Summary of the Invention

[0003] The image processing method and related apparatus provided in this application embodiment can reuse the underlying image processing chip (i.e., ISP) of the system camera to realize image algorithm processing, and has faster processing speed and better processing results.

[0004] In a first aspect, this application provides an image processing method applied to an electronic device, wherein a chip algorithm is stored in the hardware abstraction layer of the electronic device, and the method includes:

[0005] In response to a user operation on the camera control in the first application, an image is captured by the camera of the electronic device and a preview interface of the first application is displayed, wherein the first application is different from the camera application, the preview interface displays multiple shooting modes, and the preview interface includes the image captured by the camera;

[0006] In response to a user operation on a target shooting mode, the first image captured by the camera is processed based on the algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image with the shooting effect of the target shooting mode, wherein the target shooting mode is one of the multiple shooting modes selected.

[0007] The second image is displayed in the preview interface.

[0008] It can be seen that the hardware abstraction layer stores the chip algorithm corresponding to the shooting mode. Thus, when the first application starts the shooting function of the electronic device, the image data collected by the camera can be processed based on the algorithm corresponding to the target shooting mode to obtain a second image of the shooting effect of the target shooting mode. The underlying chip (i.e., the dedicated chip) of the system's camera application is reused to realize the post-processing of the image. Compared with the software algorithm (i.e., the algorithm processed by the general chip), the processing speed and processing effect can be improved.

[0009] In one possible implementation of the first aspect, after the second image is displayed in the preview interface, the method further includes:

[0010] In response to a user action on the shooting controls in the preview interface, the second image is saved.

[0011] It can be seen that the chip algorithm based on the hardware abstraction layer storage saves a second image with the shooting effect for the first application.

[0012] In one possible implementation of the first aspect, the electronic device includes an application framework layer that includes a camera extension library. The step of capturing an image via the camera of the electronic device and displaying a preview interface of the first application in response to a user operation on a camera control of a first application includes:

[0013] In response to user operations on the camera control of the first application, the camera interface is invoked to obtain a list of camera devices and the properties of the camera devices in the list of camera devices;

[0014] Call the camera extension interface to obtain the camera capabilities supported by the hardware of the electronic device from the camera extension library;

[0015] Based on the camera capabilities and the attributes of the camera device, images are captured through the camera of the electronic device and displayed as a preview interface of the first application.

[0016] In one possible implementation of the first aspect, the step of acquiring images via the camera of the electronic device based on the camera capabilities and attributes of the camera device, and displaying a preview interface of the first application, includes:

[0017] Based on the attributes of the camera device, determine whether the camera device supports the camera capabilities supported by the hardware, and thus determine the camera capabilities supported by the camera device.

[0018] Based on the camera capabilities supported by the camera device, images are captured by the camera of the electronic device and a preview interface of the first application is displayed, wherein each shooting mode displayed in the preview interface corresponds to the camera capabilities supported by the camera device.

[0019] It is understandable that a camera device includes logical cameras and physical cameras. A logical camera is an abstract device composed of multiple physical cameras (i.e., the camera module of an electronic device), which performs certain camera capabilities by simultaneously controlling multiple physical cameras. Therefore, different logical cameras can perform different camera capabilities, and the attributes of a camera device can be used to describe the camera capabilities it possesses. Furthermore, the camera extension library stores the camera capabilities supported by the hardware of the electronic device. Therefore, based on the camera capabilities and the attributes of the camera device, it can be determined whether each shooting mode is supported in the attributes reported by the underlying camera device. This allows the preview interface of the first application to display the shooting modes supported by the camera device and the unprocessed images captured by the camera. Because the shooting modes displayed in the preview interface are obtained after judgment, the processing speed and response speed for that shooting mode can be improved.

[0020] In one possible implementation of the first aspect, the step of processing the first image captured by the camera based on the algorithm corresponding to the target shooting mode in the chip algorithm, in response to a user operation on the target shooting mode, to obtain a second image with the shooting effect of the target shooting mode, includes:

[0021] In response to a user operation on a target shooting mode, the target camera device is powered on, wherein the target camera device includes a camera among the cameras that supports the camera capabilities corresponding to the target shooting mode;

[0022] Obtain the configuration parameters corresponding to the target shooting mode, and call the target camera device to acquire image data based on the configuration parameters;

[0023] Based on the algorithm corresponding to the target shooting mode in the chip algorithm, the first image reported by the target camera device is processed to obtain a second image with the shooting effect of the target shooting mode.

[0024] As can be seen, based on camera capabilities and the attributes of the camera device, it can be determined whether each shooting mode is supported by the camera device attributes reported at the underlying level, thereby determining the camera capabilities supported by the camera device and displaying the corresponding shooting mode. Therefore, when the first application selects to enter a certain shooting mode, it can select the camera device corresponding to that shooting mode to directly open the camera, thereby improving response speed.

[0025] In one possible implementation of the first aspect, the electronic device includes an image signal processor, wherein the algorithm based on the target shooting mode in the chip algorithm processes the first image reported by the target camera device to obtain a second image having the shooting effect of the target shooting mode, including:

[0026] The hardware abstraction layer activates the sensor of the target camera device to acquire image light signals and transmits the image light signals to the image signal processor;

[0027] The image signal processor processes the image light signal based on the algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image with the shooting effect of the target shooting mode.

[0028] As can be seen, the camera device corresponding to the target shooting mode is one that supports that mode. Therefore, the hardware abstraction layer can directly call the corresponding sensor to acquire image light signals. The acquired image light signals conform to the specifications corresponding to the target shooting mode, which can further improve the processing speed. Then, the image signal processor processes the acquired image light signals based on the algorithm corresponding to the target shooting mode stored in the chip algorithm. Because the image signal processor is a dedicated chip for image processing, it has a fast processing speed and good processing effect, and can quickly refresh the second image with the shooting effect in the preview interface.

[0029] In one possible implementation of the first aspect, the electronic device includes an image signal processor, wherein saving the second image in response to a user operation on a shooting control in the preview interface includes:

[0030] In response to a user operation on the shooting control in the preview interface, a photo-taking request is sent to the hardware abstraction layer, wherein the photo-taking request contains photo-taking parameters corresponding to the target shooting mode, and the photo-taking parameters are obtained from the camera extension library;

[0031] The hardware abstraction layer activates the sensor of the target camera device to acquire image light signals based on the photographing parameters;

[0032] The image signal processor processes the image light signal based on the algorithm corresponding to the target shooting mode in the chip algorithm, and saves a second image with the shooting effect of the target shooting mode.

[0033] As can be seen, the camera device corresponding to the target shooting mode is one that supports that mode. Therefore, the hardware abstraction layer can directly call the corresponding sensor to acquire image light signals. The acquired image light signals conform to the specifications corresponding to the target shooting mode, which can further improve the processing speed. Then, the image signal processor processes the acquired image light signals based on the algorithm corresponding to the target shooting mode stored in the chip algorithm. Because the image signal processor is a dedicated chip for image processing, it can improve the speed of image saving.

[0034] In one possible implementation of the first aspect, the shooting mode includes one or more of night mode, HDR mode, beauty mode, bokeh mode, and automatic mode.

[0035] Secondly, an electronic device is provided in the embodiments of this application, the electronic device comprising: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the method of drawing smooth rounded corners described in the first aspect or any possible implementation of the first aspect.

[0036] Thirdly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run a computer program or instructions to perform the method for drawing smooth rounded corners described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0037] In one possible implementation, the chip or chip system described above in the embodiments of this application further includes at least one memory, which stores instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0038] Fourthly, embodiments of this application provide a computer storage medium storing a computer program that, when executed by a processor, causes the computer to perform a method for drawing smooth rounded corners as described in the first aspect or any possible implementation thereof.

[0039] Fifthly, embodiments of this application provide a computer program product that, when run on a communication device, causes the communication device to perform a method for drawing smooth rounded corners as described in the first aspect or any possible implementation thereof.

[0040] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments.

[0041] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0042] The accompanying drawings used in the embodiments of this application are described below.

[0043] Figure 1 This is a schematic diagram of a software architecture for a third-party application to implement camera capabilities, provided in an embodiment of this application.

[0044] Figure 2 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the software architecture of the electronic device provided in the embodiments of this application;

[0046] Figures 4A-4C This application provides a set of user interfaces for third-party applications to call the camera application.

[0047] Figure 5 This is a timing diagram of an image processing method provided in an embodiment of this application;

[0048] Figure 6 This is a flowchart illustrating an image processing method provided in an embodiment of this application. Detailed Implementation

[0049] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0050] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0051] Typically, third-party applications, when accessing the camera on an electronic device, often suffer from poor image quality and limited shooting modes. Therefore, based on Google... The provided camera expansion interface implements a set of camera capability open solutions on the camera of electronic devices, which are used to open up some camera functions to third-party applications, thereby improving the photo effect and improving the photo quality.

[0052] Please see Figure 1 , Figure 1 This is a schematic diagram of a software architecture for a third-party application to implement camera capabilities, provided in an embodiment of this application. From... Figure 1 It can be seen that Google It provides a set of extended application programming interfaces (APIs), which are implemented by Original Equipment Manufacturers (OEMs) and encapsulated by the Camera2 Extension API and CameraX Extension API to provide capability interfaces. These camera capabilities include, but are not limited to: Auto, Face retouch, High Dynamic Range (HDR), Night mode, Bokeh / bokeh, and more.

[0053] like Figure 1 As shown, the application layer of the electronic device includes packages of third-party applications that can access camera permissions. These third-party applications include a third-party camera 2 application and a third-party camera X application. To implement camera capabilities, the third-party camera 2 application and / or the third-party camera X application first call the corresponding interfaces to obtain the preview stream and capture stream from the hardware abstraction layer (HAL). For example, the third-party camera 2 application can call the Camera 2 API in the application framework layer to obtain the preview stream and capture stream from the HAL. The preview stream and capture stream include the image obtained after the light sensor in the camera converts the captured light source signal into an electrical signal; this image can be called a raw image or a RAW image.

[0054] For example, a third-party Camera X application can use the Camera X API to call the Camera2 API to obtain the preview and image streams from the HAL. To enable camera capabilities, after obtaining the preview and image streams through the Camera2 API or CameraX API, the corresponding extended interface can be called to transmit the preview and image streams to the camera extension OEM library for camera algorithm processing. Finally, the processed image data is returned to the third-party Camera2 application and / or the third-party Camera X application, which can then display the processed image.

[0055] For example, the Camera2 API transmits the preview stream and the photo stream to the Camera2 extension API. The Camera2 extension API calls the camera extensions interface to transmit the preview stream and the photo stream to the camera extension OEM library in the application framework layer for image algorithm processing. The camera extension OEM library transmits the processed image data to the Camera2 extension API and finally returns it to the third-party Camera2 application.

[0056] For example, the CameraX API transmits the preview stream and the photo stream to the CameraX extension API. The CameraX extension API calls the camera extensions interface to transmit the preview stream and the photo stream to the camera extension OEM library in the application framework layer for image algorithm processing. The camera extension OEM library transmits the processed image data to the CameraX extension API and finally returns it to the third-party camera application.

[0057] It can be seen that, based on Figure 1 The software architecture shown has images returned by the hardware abstraction layer that have not undergone algorithmic processing. These images require image post-processing algorithms from the camera extension OEM library in the application framework layer to help third-party applications implement camera capabilities. This is purely software algorithm processing, which may result in slow processing speed and poor processing results.

[0058] In view of this, this application provides an image processing method applied to an electronic device with image processing capabilities. The hardware abstraction layer of this electronic device stores chip algorithms corresponding to shooting modes. Therefore, when a third-party application activates the electronic device's shooting function and selects a target shooting mode from multiple shooting modes, based on the support of the image information processor (ISP) in the hardware layer, the hardware abstraction layer can process the image data reported by the camera device based on the algorithm corresponding to the target shooting mode in the chip algorithm, obtaining an image with the desired shooting effect. This image is then displayed in the preview interface of the third-party application. It can be seen that this application reuses the underlying image processing chip (i.e., ISP) of the system camera to implement image algorithm processing, which has a faster processing speed and better processing results compared to software algorithms.

[0059] First, the electronic device in this application embodiment can be a smart screen device, a smart TV (TV), a mobile phone, a tablet computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), a wearable device (such as a smartwatch, a smart bracelet), and other devices with display functions. This application embodiment does not impose any special restrictions on the specific form of the electronic device.

[0060] For example, taking a mobile phone as an electronic device, Figure 2 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. That is, exemplary, Figure 2 The electronic device shown could be a mobile phone.

[0061] like Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 221, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0062] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0063] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0064] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0065] The processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0066] The charging management module 140 receives charging input from the charger. The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 221, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In another embodiment, the power management module 141 can also be located in the processor 110. In another embodiment, the power management module 141 and the charging management module 140 can also be located in the same device.

[0067] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0068] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In another embodiment, the antenna can be used in conjunction with a tuning switch.

[0069] The mobile communication module 150 can provide wireless communication solutions for applications on the electronic device 100, including second-generation (2G), third-generation (3G), fourth-generation (4G), fifth-generation (5G), and sixth-generation (6G) mobile communication technologies. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0070] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment, the modem processor may be a separate device. In another embodiment, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0071] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0072] In one embodiment, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0073] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] The display screen 194 is used to display images, videos, user interfaces, controls, windows, etc. In one implementation, the display screen 194 is used to display smooth rounded corners obtained by a smooth rounded corner drawing method provided in this application. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In one embodiment, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0075] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0076] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one implementation, the ISP can be integrated into the camera 193.

[0077] In one possible implementation, the ISP can also select a corresponding image algorithm to optimize the image based on the selected shooting mode, thereby obtaining the color matching effect corresponding to that shooting mode. The shooting modes include one or more of the following: night mode, HDR mode, beauty mode, bokeh mode, and automatic mode.

[0078] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0079] Camera 193 may include one or more of the following: telephoto camera, wide-angle camera, ultra-wide-angle camera, zoom camera, or depth camera, etc. Among them, a telephoto camera has a small shooting range and is suitable for shooting distant objects; a wide-angle camera has a large shooting range; an ultra-wide-angle camera has a larger shooting range than a wide-angle camera and is suitable for shooting panoramic or other large-scale scenes. A depth camera can be used to measure the distance to the object being photographed, that is, to measure the depth information of the object being photographed. For example, it may include a 3D depth-sensing camera, a time-of-flight (TOF) depth camera, or a binocular depth camera, etc.

[0080] The camera 193 may include a main camera and a secondary camera. The main camera can be used to capture images, and may include, for example, a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a zoom camera. The secondary camera can be used for ranging or other auxiliary functions, and may include, for example, a depth camera.

[0081] Camera 193 may include a front-facing camera and / or a rear-facing camera. The front-facing camera may include one or more main cameras, and the rear-facing camera may also include one or more main cameras. When capturing an image, the target main camera used by the electronic device to acquire the image can be the default main camera or a main camera selected by the user.

[0082] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0083] Internal memory 221 can be used to store computer executable program code, which includes instructions. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 221 and / or instructions stored in memory located in the processor.

[0084] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0085] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals.

[0086] The loudspeaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals.

[0087] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals.

[0088] The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0089] The 170D headphone jack is used to connect wired headphones.

[0090] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In one embodiment, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands.

[0091] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In one embodiment, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B.

[0092] The 180C barometric pressure sensor is used to measure barometric pressure.

[0093] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover.

[0094] The accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes).

[0095] Distance sensor 180F is used to measure distance.

[0096] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from a nearby object. When sufficient reflected light is detected, it can be determined that an object is near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that no object is near the electronic device 100.

[0097] The 180L ambient light sensor is used to detect ambient light intensity.

[0098] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0099] The 180J temperature sensor is used to detect temperature.

[0100] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0101] The bone conduction sensor 180M can acquire vibration signals.

[0102] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0103] Motor 191 can generate vibration alerts. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card.

[0104] The software system of electronic devices (such as mobile phones) can adopt a layered architecture, transaction-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android system as an example to illustrate the software architecture of a mobile phone. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of the software architecture of the electronic device provided in the embodiments of this application.

[0105] like Figure 3 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application frame layer, hardware abstraction layer (HAL), driver layer, and hardware layer. Wherein:

[0106] The application layer can include a series of packages. For example, applications can include third-party applications, as well as applications like camera and gallery apps. Third-party applications include those that can invoke the camera app to initiate photo taking, such as third-party Camera 2 and Camera X apps. The camera app can include, but is not limited to, a UI module, a photo taking module, and a gallery module. The UI module, such as the cameraUI module, is primarily responsible for human-computer interaction in the camera app, such as controlling the preview interface and its display, and receiving and responding to user actions within the preview interface. The photo taking module provides functions such as taking photos and focusing. The gallery module can be used to store photos taken by the user in the electronic device's file system or a specific database for retrieval by applications such as the gallery.

[0107] Among them, third-party Camera2 applications use the Camera2 API to call camera functions, while third-party CameraX applications use the CameraX interface (i.e., CameraX API) or the CameraX extension interface (i.e., CameraXextension API) to call camera functions. The CameraX API is integrated into the application layer.

[0108] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. It mainly involves the camera framework and can include camera access interfaces such as camera extension libraries and camera services. It plays a bridging role and can interact with camera applications through the application API and HAL through the HAL interface definition language (HIDL).

[0109] The camera access interface includes the Camera2 API, the Camera2extension API, and the Camera Extension API.

[0110] The camera extension library is an implementation library of open standard interfaces for camera capabilities, providing third-party applications with camera functions such as auto, beautification, bokeh, HDR, and night mode. The camera extension library is used to determine whether a request from a third-party application supports the corresponding camera capability, report the streaming resolution, report the streaming, and send the parameters required for taking the photo.

[0111] The Camera Service responds to requests from the application layer, providing session management, device management, and provider management. The Camera Service is an independent process that acts as a server, handling cross-process requests from application clients, performing internal operations, and then, as a client, retransmitting the request to the HAL process, which is acting as the server. Therefore, the Camera Service is part of the camera's pathway, playing a crucial bridging role.

[0112] The Hardware Abstraction Layer (HAL) is an interface layer located between the application framework layer and the driver layer. It provides a virtual hardware platform for the operating system. For example, the HAL receives requests from the Camera Provider, forwards the requests to the driver, waits for the results to be returned, and then reports them to the Camera Provider.

[0113] For example, the hardware abstraction layer may include a camera hardware abstraction layer and a chip algorithm library. The camera hardware abstraction layer can provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera), and other camera devices. The camera algorithm library stores various image processing algorithms. For example, in this embodiment, the camera algorithm library may include automatic algorithms, beautification algorithms, bokeh algorithms, night scene algorithms, and HDR algorithms, etc.

[0114] Automatic algorithms are used to enable the camera's automatic capability, allowing it to automatically adjust the final image based on the surrounding scenery when a third-party application calls the camera app to take a picture.

[0115] Skin-smoothing algorithms are used to implement the camera's skin-smoothing capability, which modifies facial color, contours, etc., when a third-party application calls the camera app to take still images.

[0116] Blur algorithms are used to achieve the camera's ability to blur backgrounds, making foreground subjects clearer when a third-party application uses the camera app to take a photo.

[0117] Night scene algorithms are used to enable the camera's night scene capability, allowing third-party applications to call the camera app to capture the best still images in low-light environments (such as at night).

[0118] The HDR algorithm is used to implement the HDR camera capability by using different automatic exposure (AE) settings when a third-party application calls the camera to take a picture in order to obtain the best results.

[0119] The driver layer is the layer between hardware and software, and it includes drivers for various hardware components. The driver layer can include camera drivers, digital signal processor (DSP) drivers, and image processor (IPC) drivers, among others. Specifically, camera drivers are used to drive the image sensors of one or more cameras to acquire images and to drive the IPC to preprocess the images. DSP drivers are used to drive the digital signal processor to process images. IPC drivers are used to drive the graphics processor to process images.

[0120] The hardware layer may include a camera module, an image signal processor, a digital signal processor, and an image processor. The camera may include one or more image sensors (e.g., image sensor 1, image sensor 2, etc.). Optionally, the camera may also include a camera motor, a lens, a Time-of-Flight (TOF) sensor, etc.

[0121] The following example illustrates the workflow of electronic device software and hardware, using a scenario where a third-party application calls the camera application.

[0122] In this embodiment, when the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the user operation into a raw input event (including touch coordinates, touch operation timestamp, etc.) and identifies the control corresponding to the input event. Taking a touch operation as a user operation acting on a camera control in a third-party application as an example, the third-party application calls the camera extension interface of the application framework layer to activate the camera capabilities provided in the camera extension library. Then, by calling the camera device in the camera hardware abstraction layer, such as camera device 1, it sends a command to activate the camera capabilities. The camera hardware abstraction layer sends this command to the camera driver in the driver layer. The camera driver can activate the sensor corresponding to the camera device, such as sensor 1, and then acquire image light signals through the sensor. The image light signals are then transmitted to the image signal processor for preprocessing to obtain a RAW image. Finally, the raw stream is transmitted back to the hardware abstraction layer through the camera driver.

[0123] The Hardware Abstraction Layer (HAL) sends RAW image streams to the chip's algorithm library. Supported by the image signal processor, the HAL performs algorithmic optimization on the RAW image streams, such as beautification / body shaping algorithms, automatic algorithms, night scene algorithms, HDR algorithms, and bokeh algorithms, to obtain better-looking images. The HAL then returns this image data to third-party applications via the camera extension interface. Subsequently, third-party applications and gallery applications, with the support of a window manager, can display photos with camera capabilities to the user, such as beautified photos, etc.

[0124] It can be seen that, Figure 3 The software architecture shown is for Figure 1 In the software architecture shown, the electronic device can reuse the system camera's image signal processor in the hardware abstraction layer to perform algorithmic processing on the images transmitted back by the camera. Therefore, the image returned to the camera service by the hardware abstraction layer, having undergone algorithmic processing, does not need further processing in the application framework layer and can be directly returned to the third-party application via the camera service.

[0125] Figures 4A-4C This is a set of user interfaces for third-party applications to call the camera application, provided in the embodiments of this application.

[0126] Figure 4A This is the main interface (homepage) of an electronic device provided in this application embodiment. For example... Figure 4A As shown, the main interface 113 may include a status bar 111, a page indicator 112, and multiple application icons.

[0127] The status bar 111 may include one or more signal strength indicators for mobile communication signals (also known as cellular signals), wireless fidelity (Wi-Fi) signal strength indicators, battery status indicators, time indicators, etc.

[0128] Page indicator 112 can be used to indicate the positional relationship between the currently displayed page and other pages.

[0129] Multiple application icons may include a settings application icon 113A, an app store application icon 113B, a browser application icon 113C, a third-party application icon 113D, a gallery application icon 113E, a camera application icon 113F, a contacts application icon, a phone application icon, a messaging application icon, etc. The main interface 113 may also include other application icons, which are not listed here. These multiple application icons can be distributed across multiple pages. A page indicator 112 can be used to indicate which of the multiple pages hosting the multiple applications the user is currently viewing. Users can use left and right swipe gestures to browse other pages. The third-party applications are those with camera functionality, meaning they include camera controls that can call up the camera application.

[0130] Understandable. Figure 4A The user interface described below is merely an illustrative example of a possible user interface style for an electronic device, taking a mobile phone as an example, and should not be construed as limiting the embodiments of this application.

[0131] like Figure 4A As shown, the electronic device can detect user actions performed on the third-party application icon 113D, and in response to these actions, the electronic device can display the following: Figure 4B The user interface shown. Figure 4B This is the main interface of a third-party application provided in this application embodiment. For example... Figure 4B As shown, the main interface 114 may include a camera control 1141, which is used to invoke the camera application of the electronic device to take pictures in a third application. The electronic device can detect user operations on the camera control 1141, and in response to the operations, the electronic device can display as shown in the image. Figure 4C The user interface shown. Figure 4C This is a user interface, also known as a preview interface, provided in this application embodiment, for a third-party application to call the shooting and display services provided by a camera application.

[0132] like Figure 4C As shown, the preview interface may include a mode bar 121, a shooting control 122, a preview window 123, and a flip control 124.

[0133] The mode bar 121 may include multiple shooting mode options, which indicate the camera capabilities supported by the electronic device for third-party applications. These shooting mode options include, but are not limited to, automatic mode option 1211, night scene mode option 1212, beauty mode option 1213, HDR mode option 1214, bokeh mode option 1215, etc. Different shooting modes can provide users with different shooting effects, and users can select any of these shooting modes from the multiple options in the third-party application according to their different needs. It is understood that the selection of one or more shooting modes can be represented on the interface as text information, such as "automatic," "night scene," "beauty," "HDR," and "bokeh," or as images or other forms of interactive elements; this application does not limit this.

[0134] The electronic device can detect user actions on the mode bar 121 and change the currently used shooting mode according to the user actions. Such user actions include, for example, left / right swipes. For instance, when the device detects a left swipe (left swipe) on the mode bar 121 and the cursor 125 stops at the "Blur" option, the electronic device can switch to "Blur" mode. By default, the electronic device 100 first uses the normal mode with no photo effects.

[0135] The shooting control 122 is used to trigger taking a picture. The electronic device can detect whether there is a user operation on the shooting control 122, such as a click operation. When a user operation on the shooting control 122 is detected, the electronic device 100 can generate a picture-taking command. The electronic device 100 can acquire the image reported by the camera with the corresponding timestamp according to the picture-taking command, and then save it as a photo. In summary, the user operation on the shooting control 122 is the shooting operation described in this application.

[0136] Preview window 123 can be used to display images reported by the camera in real time. In different shooting modes, electronic device 100 can process the images reported by the camera based on chip algorithms stored in the hardware abstraction layer, improving the image display effect. For example, from... Figure 4C As can be seen, in "blur" mode, the electronic device 100 can blur the background in the image reported by the camera to highlight the foreground person, and the preview window 123 of the third-party application can display the image with the "blur" effect. Figure 4C (The "bokeh" effect is not shown). Understandably, before a shooting mode is selected, the preview window 123 displays the default image reported by the camera, which is an image that has not undergone algorithmic optimization.

[0137] The flip control 124 can be used to listen for user operations that trigger the flip camera. The electronic device can detect user operations on the flip control 124, such as touch operations. In response to this operation, the electronic device can switch the camera used for shooting, for example, switching the rear camera to the front camera, or vice versa. Different cameras support different shooting modes; that is, the front camera or the rear camera can support one or more of the above shooting modes. For specific support capabilities, please refer to the following... Figure 5 This will not be elaborated upon here.

[0138] Understandably, the preview window 123 can display images processed by image processing algorithms corresponding to different shooting modes in real time, so that users can perceive the shooting effects corresponding to different shooting modes in real time, thereby improving the shooting effects and shooting quality of third-party applications.

[0139] For example, when a user operation is detected on the automatic mode option 1211, the electronic device can switch to "automatic" mode. In the hardware abstraction layer, the image captured by the camera is processed based on the image processing algorithm corresponding to automatic mode, and the final image is automatically adjusted according to the surrounding scenery. The image processed by the image processing algorithm corresponding to automatic mode is displayed in real time in the preview window 123, allowing the user to experience the shooting effect of automatic mode in real time. When a user operation is detected on the shooting control 122, the electronic device can generate a shooting command. The electronic device can then acquire the image corresponding to automatic mode according to the shooting command and save it as a photo.

[0140] When a user operation is detected on the Night Scene mode option 1212, the electronic device can switch to "Night Scene" mode. In the hardware abstraction layer, the image captured by the camera is processed based on the image processing algorithm corresponding to Night Scene mode, thereby adjusting the brightness to adapt to low-light environments. The image processed by the image processing algorithm corresponding to Night Scene mode is displayed in real time in the preview window 123, allowing the user to experience the shooting effect of Night Scene mode in real time. When a user operation is detected on the shooting control 122, the electronic device can generate a shooting command. The electronic device can then acquire the image corresponding to Night Scene mode according to the shooting command and save it as a photo.

[0141] When a user operation is detected on the beautification mode option 1213, the electronic device can switch to "beautification" mode. In the hardware abstraction layer, the image captured by the camera is processed based on the image processing algorithm corresponding to the beautification mode, thereby adjusting and refining the facial color, contours, etc. The image processed by the image processing algorithm corresponding to the beautification mode is displayed in real time in the preview window 123, allowing the user to experience the shooting effect of the beautification mode in real time. When a user operation is detected on the shooting control 122, the electronic device can generate a shooting command. The electronic device can then acquire the image corresponding to the beautification mode according to the shooting command and save it as a photo.

[0142] When a user operation is detected on the HDR mode option 1214, the electronic device can switch to "HDR" mode. In the hardware abstraction layer, the image captured by the camera is processed based on the image processing algorithm corresponding to the HDR mode, thereby using different automatic exposure configurations to obtain the best effect. The image processed by the image processing algorithm corresponding to the HDR mode is displayed in real time in the preview window 123, allowing the user to experience the shooting effect of the HDR mode in real time. When a user operation is detected on the shooting control 122, the electronic device can generate a shooting command. The electronic device can then acquire the image corresponding to the HDR mode according to the shooting command and save it as a photo.

[0143] Based on the hardware and software architecture of the aforementioned electronic device, this application introduces an image processing method.

[0144] Figure 5 This is a timing diagram of an image processing method provided in an embodiment of this application. The implementation of this method can be based on the interaction and cooperation between the application layer (such as a third-party application), the application framework layer, and the hardware abstraction layer (HAL) within an electronic device, such as... Figure 5 As shown, the method may include:

[0145] Third-party applications accessing the camera

[0146] S1, third-party applications can launch the camera function.

[0147] Specifically, the system desktop can detect when a user opens a third-party application. For example, if the user clicks on the desktop image of a "third-party application," the main interface of the third-party application will be displayed. This main interface includes a camera control for accessing camera functions. The electronic device detects when the user activates the camera control; for example, if the user clicks on the "camera control" interface element, a preview of the third-party application will be displayed. This preview interface can be... Figure 4C The display includes a mode bar 121, shooting controls 122, and a preview window 123, etc.

[0148] S2, third-party applications request a list of camera devices from the camera service.

[0149] Specifically, when a third-party application activates the camera function, it can call a camera interface (such as the Camera2 API or CameraX API) to obtain a list of available camera devices (Camera IDs). This list stores all camera IDs owned by the electronic device. Each camera ID includes a physical camera ID and a logical camera ID. A logical camera is an abstract device composed of multiple physical cameras. A logical camera performs certain camera capabilities by simultaneously controlling multiple physical camera devices. The logical camera ID is a unique string representing the abstract capabilities of multiple physical cameras.

[0150] S3, the camera service returns a list of camera devices to third-party applications.

[0151] Specifically, the camera service calls the camera interface to return a list of camera devices to third-party applications.

[0152] S4, third-party applications request camera attributes supported by the camera device from the camera service.

[0153] Specifically, third-party applications can call the camera interface for each Camera ID to retrieve its supported set of camera attributes (get Camera Characteristics). Each camera device's attributes are composed of key-value pairs. The key, also called an attribute tag, is used to retrieve the corresponding attribute value, and the value describes the specific numerical value of the capability. The value can be an integer or an array. Camera Characteristics is the attribute class describing the camera device, and its attributes are fixed.

[0154] S5, the camera service returns the camera properties supported by the camera device to third-party applications.

[0155] Specifically, the camera service calls the camera interface to return the camera properties supported by the camera device to third-party applications.

[0156] In S6, third-party applications request underlying camera capabilities supported by the camera extension library.

[0157] Specifically, the third application can call camera extension interfaces (such as the Camera2 extension API or CameraX extension API) to obtain the camera capabilities supported by the underlying (i.e., the hardware of the electronic device) from the camera extension library. This camera extension interface sequentially checks whether each camera capability is supported in the reported Camera Characteristics by calling the (is Extension Available) interface. Finally, the (isExtension Available) interface can determine the supported camera capabilities for each Camera ID from the set of camera capabilities (CameraCharacteristics) corresponding to each Camera ID. These camera capabilities include, but are not limited to: Auto, Beauty, Bokeh (Portrait), HDR, and Night Mode, etc.

[0158] Generally speaking, only one Camera ID on the rear camera supports all camera capabilities.

[0159] In one implementation, the third-party application calls the `isExtensionAvailable` interface to determine whether a given Camera ID supports a certain camera capability, thus enabling a specific shooting mode. The parameters passed from the third-party application to the camera extension library include the Camera ID, the set of camera attributes supported by the Camera ID, and the underlying supported camera capabilities.

[0160] For example, the first tag can be the underlying supported camera capabilities, such as com.hihonor.device.capabilities.extensionType. The value corresponding to the first tag can be an array collection, such as [0,1,2,3,4], where 0 represents the automatic camera capability, 1 represents the beautification camera capability, 2 represents the bokeh / portrait camera capability, 3 represents the HDR camera capability, and 4 represents the night scene camera capability. Therefore, the array collection [0,1,2,3,4] can indicate that the Camera ID supports camera capabilities such as automatic, beautification, bokeh (portrait), HDR, and night scene.

[0161] For example, the first tag can be a camera device attribute, such as com.hihonor.device.capabilities.cameraIdCustomInfo. The value corresponding to the second tag can also be an array collection, representing the scene modes supported by the Camera ID through a series of rules. For example, a scene mode of 0 represents the normal shooting mode path, and the corresponding camera capabilities (i.e., extension types) are Auto, HDR, and Night; a scene mode of 23 represents the portrait shooting mode path, and the corresponding camera capabilities (i.e., extension types) are Beauty and Portrait.

[0162] Therefore, if the first label indicates that the Camera ID supports a certain camera capability, and the second label indicates that the Camera ID supports the scene mode corresponding to that camera capability, then the Camera ID supports that camera capability. Taking a rear camera as an example, Camera ID 5 supports scene mode 0. Camera ID 5 also has camera capabilities such as Auto, HDR, and Night mode. Therefore, when a third-party application enters one of these modes, it will select Camera ID 5 to open the camera.

[0163] S7's camera extension library returns camera capabilities supported by the camera device to third-party applications.

[0164] Specifically, the camera extension library calls the camera extension interface to return the camera capabilities supported by the camera device to third-party applications.

[0165] Third-party application enters shooting mode

[0166] The S8 uses a third-party application to determine the shooting mode.

[0167] Specifically, the preview interface of a third-party application can display various shooting mode options supported by the camera device, such as: Auto, Beauty, Bokeh (Portrait), HDR, and Night Mode. The third-party application can detect user actions applied to a certain shooting mode and change the currently used shooting mode accordingly.

[0168] When a third-party application enters a certain extended shooting mode, it goes through the process of opening the camera, streaming, and previewing. "Opening the camera" means opening the Camera ID device corresponding to the shooting mode, which powers on the corresponding device at the underlying level, such as powering on the device's camera sensor.

[0169] Streaming configuration typically involves configuring a preview stream and a photo stream. This can be understood as creating two separate paths for previewing and taking photos, used for sending requests and returning data.

[0170] S9: Third-party applications request configuration parameters corresponding to the shooting mode from the camera extension library.

[0171] Specifically, before the third application sends the start streaming message to the hardware abstraction layer, the third-party application can call the camera extension interface to obtain the configuration parameters corresponding to the selected shooting mode from the camera extension library. These configuration parameters include the resolution of the images transmitted in the preview stream and the capture stream, the parameters required for streaming, and so on.

[0172] For example, the camera extension interface obtains the streaming resolution from the camera extension library through the getSupportedResolutions method, that is, it obtains the resolution supported by the underlying layer from the set of camera characteristics reported by the underlying layer, and then selects the correct resolution from it through a certain algorithm.

[0173] For example, the camera extension interface obtains the streaming parameters from the camera extension library through the onPresetSession method. The streaming parameters include private parameters and public parameters corresponding to the shooting mode.

[0174] In the S10, the camera extension library returns configuration parameters corresponding to the shooting mode to third-party applications.

[0175] The configuration parameters corresponding to the shooting mode include the streaming resolution and streaming parameters. The streaming parameters include both private and public parameters specific to the shooting mode. The camera extension library calls the camera extension interface to return the configuration parameters corresponding to the shooting mode to the third-party application.

[0176] S11, third-party applications send preview requests corresponding to the shooting mode to the hardware abstraction layer.

[0177] The preview request carries the streaming resolution and streaming parameters corresponding to the preview stream. These streaming parameters include both private and public parameters specific to the shooting mode. The third-party application calls the camera interface to send a preview request corresponding to the shooting mode to the camera service, which in turn sends the preview request to the hardware abstraction layer.

[0178] For example, a third-party application can send a message to the Hardware Abstraction Layer (HAL) to initiate a streaming configuration by calling the createCaptureSession interface. The streaming configuration can include the HAL configuring the preview stream using the streaming resolution and streaming parameters.

[0179] S12, the hardware abstraction layer processes the reported image based on the chip algorithm library to obtain the shooting effect corresponding to the shooting mode.

[0180] Specifically, after receiving a message triggering the start of image acquisition, the Hardware Abstraction Layer (HAL) can send a command to the camera device to initiate image acquisition. Once the camera device starts acquiring images, the HAL receives the image stream from the camera device; this image stream can be referred to as the raw image stream. The HAL can process and compress the raw image stream from the camera device to obtain a preview stream. Furthermore, the preview stream can be configured using the acquisition resolution and acquisition parameters passed from a third-party application. These acquisition parameters include private and public parameters corresponding to the shooting mode, and the chip's algorithm library stores image algorithms corresponding to various shooting modes. The HAL can determine the image algorithm corresponding to the shooting mode based on the private parameters of that shooting mode. With the support of the image information processor, it performs algorithmic processing on the image in the preview stream to obtain the shooting effect corresponding to the shooting mode.

[0181] S13, the Hardware Abstraction Layer returns images with shooting effects to third-party applications.

[0182] Specifically, the hardware abstraction layer returns an image with the shooting effect to the camera service, and the camera service calls the camera interface to return an image with the shooting effect to a third-party application.

[0183] S14: Third-party applications display images with shooting effects in the preview window.

[0184] The preview image in the preview window displays the shooting effect corresponding to the shooting mode and can be continuously refreshed with the preview stream.

[0185] It is understandable that the essence of previewing is to continuously send request requests to the hardware abstraction layer, and then the hardware abstraction layer continuously returns images, while a request request is sent only once when taking a picture.

[0186] S15: Third-party applications request shooting parameters from the camera extension library.

[0187] Specifically, the third-party application can detect the user's photo-taking action, such as when the user presses the "take photo" control in the preview interface of the third-party application, thus triggering the photo capture. Before sending the photo capture request, the third-party application first calls the camera extension interface to obtain the photo capture parameters required to send the photo capture request from the camera extension library. These photo capture parameters include the streaming resolution and streaming parameters corresponding to the capture stream. The streaming parameters include private and public parameters corresponding to the shooting mode. For example, the third-party application calls the getCaptureStages interface to obtain the photo capture parameters required to send the photo capture request from the camera extension library.

[0188] S16, the camera extension library returns shooting parameters to third-party applications.

[0189] Specifically, the camera extension library calls the camera extension interface to return shooting parameters to third-party applications.

[0190] S17, third-party applications send photo requests corresponding to the shooting mode to the hardware abstraction layer.

[0191] The photo-taking request carries the corresponding stream resolution and stream parameters, as well as the photo-taking timestamp. The stream parameters include both private and public parameters corresponding to the shooting mode. The third-party application calls the camera interface to send the photo-taking request corresponding to the shooting mode to the camera service, and the camera service then sends the photo-taking request corresponding to the shooting mode to the hardware abstraction layer.

[0192] Regarding the `getCaptureStages` interface, this interface encapsulates the private parameters needed for taking a picture and reports them to a third-party application. The third-party application then uses these private and public parameters as the parameters for its picture-taking request and sends them to the Hardware Abstraction Layer (HAL), allowing the HAL to understand what processing is required for this frame of image. For third-party applications that do not extend camera capabilities, they can only obtain public parameters to send picture-taking requests. Therefore, they cannot use the underlying algorithms to process the image, and thus lack the corresponding shooting effects for various shooting modes. Below are some examples of parameters:

[0193]

[0194] S18, the hardware abstraction layer processes the image corresponding to the photo request based on the chip algorithm library to obtain the shooting effect corresponding to the shooting mode.

[0195] Specifically, the Hardware Abstraction Layer (HAL) receives image streams from the camera device, referred to as the raw image stream. The HAL processes and compresses these raw image streams to obtain a new image stream. Furthermore, it can configure the image stream corresponding to a given capture timestamp using the streaming resolution and streaming parameters passed from a third-party application. These streaming parameters include private and public parameters specific to the shooting mode, and the chip's algorithm library stores image algorithms for various shooting modes. The HAL determines the corresponding image algorithm for a given shooting mode based on its private parameters. With the support of the image information processor, it processes the image stream corresponding to the timestamp using the algorithm to obtain the shooting effect for that mode and generates a thumbnail of the image.

[0196] S19, the hardware abstraction layer returns the image with the shooting effect corresponding to the photo-taking request to the third-party application.

[0197] Specifically, the hardware abstraction layer returns an image with the captured effect and a thumbnail of the image to the camera service, and the camera service calls the camera interface to return an image with the captured effect and a thumbnail of the image to a third-party application.

[0198] S20 allows third-party applications to save images with shooting effects.

[0199] Please see Figure 6 , Figure 6 This is a flowchart illustrating an image processing method provided in an embodiment of this application, applied to... Figure 2 The hardware architecture shown and Figure 3 The software architecture shown includes, but is not limited to, the following steps:

[0200] Step S601: In response to a user operation on the camera control in the first application, an image is captured by the camera of the electronic device and a preview interface of the first application is displayed.

[0201] The first application can be a third-party application with a shooting function. The camera control is used to invoke the camera application of the electronic device, thereby capturing images through the electronic device's camera and displaying a preview interface of the first application. The preview interface displays various shooting modes and images captured by the camera.

[0202] For example, shooting modes include one or more of the following: Night mode, HDR mode, Beauty mode, Bokeh mode, and Auto mode. For instance... Figure 4C The shooting modes shown in mode bar 121 are as follows.

[0203] In one possible implementation, the electronic device includes an application framework layer, which includes a camera extension library, as detailed in the reference. Figure 3 The description of the camera extension library is as follows: In response to user operations on the camera control of the first application, the system calls the camera interface to obtain a list of camera devices and the properties of the camera devices in the list; it calls the camera extension interface to obtain the camera capabilities supported by the electronic device's hardware from the camera extension library; based on the camera capabilities and the camera device properties, it captures an image through the electronic device's camera and displays a preview interface of the first application. The image captured by the camera displayed in the preview interface at this time can be an image captured by the default camera without algorithmic processing.

[0204] In one implementation, the camera device's attributes are used to determine whether it supports the camera capabilities supported by the hardware, thereby determining the camera capabilities supported by the camera device. Based on the camera capabilities supported by the camera device, images are captured through the electronic device's camera and a preview interface of a third-party application is displayed. Each shooting mode displayed in the preview interface corresponds to a camera capability supported by the camera device.

[0205] For a detailed description of step S601, please refer to [link / reference]. Figure 5 S1-S7 are shown.

[0206] Step S602: In response to the user operation on the target shooting mode, the first image captured by the camera is processed based on the algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image with the shooting effect of the target shooting mode.

[0207] In one possible implementation, in response to a user operation on the target shooting mode, the target camera device is powered on. The target camera device includes a camera that supports the camera capabilities corresponding to the target shooting mode. Configuration parameters corresponding to the target shooting mode are acquired, and image data is acquired by the target camera device based on these parameters. The first image reported by the target camera device is processed using an algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image with the shooting effect of the target shooting mode. The target shooting mode is one selected from multiple shooting modes, i.e., the shooting mode that detects the user operation.

[0208] In one implementation, the electronic device includes an image signal processor. The hardware abstraction layer activates the sensor of the target camera device to acquire image light signals and transmits the image light signals to the image signal processor. The image signal processor processes the image light signals based on the algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image with the shooting effect of the target shooting mode.

[0209] For a detailed description of step S602, please refer to [link / reference]. Figure 5 S8-S12 are shown.

[0210] Step S603: Display the second image in the preview interface.

[0211] For a detailed description of step S603, please refer to [link / reference]. Figure 5 S13-S14 are shown.

[0212] After step S603, the method further includes: saving the second image in response to a user operation on the shooting control in the preview interface.

[0213] For example, in response to a user operation on the shooting control in the preview interface, a photo-taking request is sent to the hardware abstraction layer. The photo-taking request contains shooting parameters corresponding to the target shooting mode, which are obtained from the camera extension library. The hardware abstraction layer activates the sensor of the target camera device based on the shooting parameters to acquire image light signals. The image signal processor processes the image light signals based on the algorithm corresponding to the target shooting mode in the chip algorithm, and saves a second image with the shooting effect of the target shooting mode. For a detailed description, please refer to [link to relevant documentation]. Figure 5 S15-S20 are shown.

[0214] The term "user interface (UI)" used in the specification, claims, and drawings of this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a webpage, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a webpage is also defined through tags or nodes in the webpage source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>To define the elements and attributes of a webpage.

[0215] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0216] It should be understood that the steps in the above-described method embodiments provided in this application can be implemented by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0217] This application also provides an electronic device that may include a memory and a processor. The memory may be used to store a computer program; the processor may be used to invoke the computer program in the memory to cause the electronic device to perform the methods in any of the above embodiments.

[0218] This application also provides a chip system including at least one processor for implementing the functions involved in the methods performed by the electronic device in any of the above embodiments.

[0219] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0220] The chip system can consist of chips or include chips and other discrete components.

[0221] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0222] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0223] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0224] This application also provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method executed by the electronic device in any of the above embodiments.

[0225] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed by the electronic device in any of the above embodiments.

[0226] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0227] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0228] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0229] In summary, the above description is merely an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made according to the disclosure of the present invention should be included within the scope of protection of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, which includes an application layer, an application framework layer, a hardware abstraction layer, and an image signal processor. The application layer includes a first application and a camera application. The application framework layer includes a camera extension library. The hardware abstraction layer stores chip algorithms. The first application differs from the camera application and is a third-party application. In response to user operations on the camera control in the first application, the camera interface is called to obtain a list of camera devices and the camera attributes corresponding to each camera device in the list, and the camera extension interface is called to obtain the camera capabilities supported by the image signal processor from the camera extension library; the camera device list includes multiple camera devices; Based on the attributes of the camera device, determine whether the camera device supports the camera capabilities supported by the image signal processor, and thus determine the camera capabilities supported by the camera device; After determining the camera capabilities supported by the camera device, images are captured through the camera and a preview interface of the first application is displayed. The preview interface displays multiple shooting modes, each of which corresponds to the camera capabilities supported by the camera device. The preview interface includes images captured by the camera. In response to a user operation on a target shooting mode, the camera extension library returns configuration parameters corresponding to the target shooting mode to the first application; if the first application has extended camera capabilities, the configuration parameters include public parameters and private parameters; if the first application does not have extended camera capabilities, the configuration parameters include the public parameters but do not include the private parameters. The hardware abstraction layer confirms whether the private parameter exists in the configuration parameters. If the private parameter exists, the hardware abstraction layer determines the algorithm corresponding to the target shooting mode in the chip algorithm according to the private parameter, processes the first image captured by the camera based on the algorithm corresponding to the target shooting mode, obtains a second image with the shooting effect of the target shooting mode, and displays the second image in the preview interface. The target shooting mode is one of the multiple shooting modes selected. If the private parameter does not exist, the first application cannot use the algorithm corresponding to the private parameter to process the image, and displays a third image in the preview interface. The third image does not have the shooting effect corresponding to the shooting mode.

2. The method according to claim 1, characterized in that, After displaying the second image in the preview interface, the method further includes: In response to a user action on the shooting controls in the preview interface, the second image is saved.

3. The method according to claim 1 or 2, characterized in that, The step of responding to a user operation on a target shooting mode by processing the first image captured by the camera based on an algorithm corresponding to the target shooting mode to obtain a second image with the shooting effect of the target shooting mode includes: In response to a user operation on a target shooting mode, the target camera device is powered on, wherein the target camera device includes a camera among the cameras that supports the camera capabilities corresponding to the target shooting mode; Obtain the configuration parameters corresponding to the target shooting mode, and call the target camera device to acquire image data based on the configuration parameters; Based on the algorithm corresponding to the target shooting mode, the first image reported by the target camera device is processed to obtain a second image with the shooting effect of the target shooting mode.

4. The method according to claim 3, characterized in that, The algorithm based on the target shooting mode processes the first image reported by the target camera device to obtain a second image with the shooting effect of the target shooting mode, including: The hardware abstraction layer activates the sensor of the target camera device to acquire image light signals and transmits the image light signals to the image signal processor; The image signal processor processes the image light signal based on the algorithm corresponding to the target shooting mode in the chip algorithm to obtain a second image with the shooting effect of the target shooting mode.

5. The method according to claim 2, characterized in that, Saving the second image in response to a user operation on the shooting control in the preview interface includes: In response to a user operation on the shooting control in the preview interface, a photo-taking request is sent to the hardware abstraction layer, wherein the photo-taking request contains photo-taking parameters corresponding to the target shooting mode, and the photo-taking parameters are obtained from the camera extension library; The hardware abstraction layer activates the camera's sensor to collect image light signals based on the captured parameters; The image signal processor processes the image light signal based on the configuration parameters and saves a second image with the shooting effect of the target shooting mode.

6. The method according to claim 1 or 2, characterized in that, The shooting modes include one or more of the following: Night Mode, HDR Mode, Beauty Mode, Bokeh Mode, and Auto Mode.

7. An electronic device, characterized in that, The electronic device includes: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-6.

8. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-6.

9. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-6.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-6.

Images