Image processing method, chip system and electronic equipment

Abstract

The invention provides an image processing method, a chip system and electronic equipment, and relates to the technical field of electronics. The method comprises the following steps: starting a camera application in response to user operation, and obtaining a first frame image collected by a camera; the first frame image is determined to be output according to the image gain of the first frame image and the jitter amount collected by the gyroscope sensor, and the image gain is used for representing the brightness increased for the first frame image; and displaying the first frame image in a preview interface of the camera application. In a dark light environment, if a shot picture is in a jitter state, a single-frame output alignment mode can be used, so that ghosting caused by multi-frame superposition can be avoided, and the definition of a dark light preview stream is improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method, a chip system and an electronic device. Background Art

[0002] With the development of electronic devices, shooting with electronic devices has become the most common way for people. As the demand for high-resolution images continues to grow, the pursuit of high-quality visual experience is also getting higher and higher. In order to improve the image quality, the collected images are usually subjected to noise reduction processing to improve the image clarity. However, in the process of image noise reduction processing, abnormal areas may appear on the processed images. The abnormal areas are called ghost images. The existence of ghost images will affect the image quality and reduce the user experience. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, a chip system, and an electronic device. In a dark environment, if the captured image is in a shaking state, a single-frame output noise reduction method can be used to avoid ghost images introduced by superimposing multiple frames and improve the clarity of the dark-light preview stream.

[0004] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a camera and a gyroscope sensor, and the method includes:

[0005] In response to a user operation, a camera application is started to obtain a first frame of image captured by the camera;

[0006] Determine outputting the first frame of image according to an image gain of the first frame of image and a gyroscope jitter amount collected by the gyroscope sensor, wherein the image gain is used to represent brightness added to the first frame of image;

[0007] The first frame image is displayed in a preview interface of the camera application.

[0008] It can be seen that image gain can be expressed as the brightness added to the image captured by the camera, and the gyroscope jitter can indicate the jitter condition. Therefore, based on these two parameters, the electronic device can determine whether multiple frames can be aligned. If not, a single frame is output to the subsequent module for noise reduction. Compared with the noise reduction method of multi-frame fusion, ghosting can be avoided and the image quality can be improved.

[0009] In a possible implementation manner of the first aspect, determining to output the first frame of image according to an image gain of the first frame of image and a jitter amount collected by the gyroscope sensor includes:

[0010] When determining that the shooting environment is a dark light environment according to the image gain of the first frame image based on a brightness algorithm, obtaining a gyroscope jitter amount collected by the gyroscope sensor;

[0011] When it is determined based on the jitter algorithm according to the jitter amount of the gyroscope that the captured image is in a jitter state, it is determined to output the first frame image.

[0012] It can be seen that the electronic device can determine that the shooting environment is a dark light environment based on the built-in brightness algorithm, and can determine that the shot picture is in a shaking state based on the built-in jitter algorithm. Because the present application has found through research that in a dark light environment, because the number of feature points on the image is small, it is impossible to use a feature point-based method for alignment, and thus an alignment method based on gyroscope matching and image superposition will be used. Because the shot picture is shaking, the picture content will be different, and image fusion will cause ghosting. Therefore, in a dark light environment, if the shot picture shakes, the embodiment of the present application can directly output a single frame to the subsequent module, thereby avoiding ghosting introduced by gyroscope-based image superposition.

[0013] In a possible implementation of the first aspect, determining that the shooting environment is a dark light environment according to the image gain of the first frame image based on a brightness algorithm includes:

[0014] When it is determined based on the brightness algorithm that the image gain of the first frame image is greater than or equal to a preset threshold, it is determined that the shooting environment is a dark light environment.

[0015] Among them, the preset threshold is the gain threshold. The camera, as the "eyes" of the electronic device, needs to constantly adjust parameters to adapt to different ambient brightness. The gain can reflect the ambient brightness, so that the shooting environment at this time can be determined more accurately.

[0016] In a possible implementation of the first aspect, determining that the captured picture is in a shaking state according to the gyroscope shaking amount based on a shaking algorithm includes:

[0017] When it is determined based on the jitter algorithm that the jitter amount of the gyroscope is greater than or equal to the optical image stabilization OIS value, it is determined that the captured image is in a jitter state, wherein the OIS value is used to indicate the displacement amount caused by vibration that can be compensated by the electronic device.

[0018] It can be seen that the optical image stabilization OIS value is expressed as the angle that can be rotated and / or the distance that can be moved. When the gyroscope shake exceeds this angle and / or distance, the optical image stabilization cannot be compensated, which will cause the captured image to be in an unstable state. Based on the OIS value, the stability of the captured image can be determined more accurately.

[0019] In a possible implementation of the first aspect, the method further includes:

[0020] When it is determined that the shooting environment is a bright light environment based on the brightness algorithm according to the image gain of the first frame of image, or when it is determined that the shooting picture is in a stationary state based on the jitter algorithm according to the jitter amount of the gyroscope, a second frame of image is acquired, where the second frame of image includes an image captured before the first frame of image or stored in a cache;

[0021] Determine a third frame image according to the first frame image and the second frame image;

[0022] The third frame image is outputted, and the third frame image is displayed in a preview interface of the camera application.

[0023] In one implementation, when the image gain of the first frame image is determined to be less than a preset threshold value based on the brightness algorithm, the shooting environment is determined to be a bright light environment. The equal situation can also be placed in another branch of the judgment, for example, when the image gain of the first frame image is determined to be less than or equal to the preset threshold value based on the brightness algorithm, the shooting environment is determined to be a bright light environment.

[0024] In one implementation, when it is determined based on the jitter algorithm that the jitter amount of the gyroscope is less than the optical image stabilization OIS value, it is determined that the captured image is in a stationary state. The equal situation can also be placed in another branch of the judgment, for example, when it is determined based on the jitter algorithm that the jitter amount of the gyroscope is less than or equal to the optical image stabilization OIS value, it is determined that the captured image is in a stationary state.

[0025] It can be seen that in bright light environments, there are many feature points on the image, and the alignment method based on feature points can bring good noise reduction effects in the subsequent multi-frame fusion processing. In dark light environments, for static images, the alignment method based on image superposition can bring good noise reduction effects in subsequent processing. Therefore, this application adds additional mode judgment conditions on the basis of multi-frame fusion / matching, which can effectively improve the clarity of dark light video streams.

[0026] In a possible implementation of the first aspect, the electronic device includes a hardware abstraction layer HAL, and the brightness algorithm and the dithering algorithm are stored in the HAL.

[0027] It can be seen that with the updating and improvement of technology, the algorithms stored in HAL can be updated to adapt to different application scenarios, making the application convenient and efficient.

[0028] In a second aspect, the present application provides a chip or a chip system, the chip or chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the image processing method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0029] In a possible implementation, the chip or chip system described above in the embodiment of the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0030] In one implementation, the chip or chip system includes an image signal processor ISP;

[0031] The ISP is used to receive a first frame of image captured by a camera of the electronic device;

[0032] The ISP is used to determine the output of the first frame image according to the image gain of the first frame image and the jitter amount collected by the gyroscope sensor, wherein the image gain is used to represent the brightness increased for the first frame image.

[0033] In a possible implementation manner of the second aspect, the ISP is specifically configured to:

[0034] When determining that the shooting environment is a dark light environment according to the image gain of the first frame image based on a brightness algorithm, obtaining a gyroscope jitter amount collected by the gyroscope sensor;

[0035] When it is determined based on the jitter algorithm according to the jitter amount of the gyroscope that the captured image is in a jitter state, it is determined to output the first frame image.

[0036] In a possible implementation manner of the second aspect, the ISP is specifically configured to:

[0037] When it is determined based on the brightness algorithm that the image gain of the first frame image is greater than or equal to a preset threshold, it is determined that the shooting environment is a dark light environment.

[0038] In a possible implementation manner of the second aspect, the ISP is specifically configured to:

[0039] When it is determined based on the jitter algorithm that the jitter amount of the gyroscope is greater than or equal to the optical image stabilization OIS value, it is determined that the captured image is in a jitter state, wherein the OIS value is used to indicate the displacement amount caused by vibration that can be compensated by the electronic device.

[0040] In a third aspect, an electronic device is provided by an embodiment of the present application, the electronic device comprising: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image processing method described in the first aspect or any possible implementation of the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the computer executes the image processing method described in the first aspect or any possible implementation of the first aspect.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a communication device, enables the communication device to execute an image processing method as described in the first aspect or any possible implementation of the first aspect.

[0043] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments.

[0044] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The following is an introduction to the drawings used in the embodiments of the present application.

[0046] Figure 1A is a schematic diagram of an image alignment process provided by an embodiment of the present application;

[0047] Figure 1B is a schematic diagram of a ghost provided in an embodiment of the present application;

[0048] Figure 2 The hardware structure of the electronic device of the embodiment of the present application is exemplarily shown;

[0049] Figure 3 is a schematic diagram of the software architecture of the electronic device provided in the embodiment of the present application;

[0050] Figure 4A-4D A user interface of a group of camera applications provided in an embodiment of the present application;

[0051] Figure 5 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0052] Figure 6 It is a schematic diagram of a flow chart of determining an alignment method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0054] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0055] In order to clearly describe the technical solutions of the embodiments of the present application, some terms involved in the embodiments of the present application are briefly introduced below.

[0056] 1Brightness(luminance)

[0057] The unit of brightness is candela per square meter (cd / square meter), which reflects the intensity of light emitted / reflected from the surface of an object and is used to emphasize the degree of perception by the human eye.

[0058] Image brightness refers to the brightness level of the image captured by the camera. The camera usually automatically adjusts the exposure time and aperture size according to the intensity of the ambient light to ensure that the captured image has appropriate brightness. In a darker environment, the camera will increase the exposure time and expand the aperture to increase the brightness of the image; in a brighter environment, the camera will reduce the exposure time and narrow the aperture to reduce the brightness of the image.

[0059] 2. Image Noise

[0060] Image noise is the random signal interference that an image is subject to when it is captured or transmitted, manifested as random changes in image information or pixel brightness, and unnecessary or redundant interference information in the image data. An image usually contains a lot of noise, such as some random, discrete, and isolated pixels caused by factors such as imaging sensor noise, photo grain noise, and channel transmission errors during image transmission. Among them, the presence of noise seriously affects the image quality.

[0061] For example, when the shooting scene of the electronic device is a dark light scene (also called a low-brightness scene), the image displayed by the electronic device is relatively dark, which affects the display effect. In order to increase the brightness of the displayed image, the sensitivity of the image sensor can be increased so that the brightness of the image reaches normal brightness. This method can improve the brightness of the image captured by the electronic device to a certain extent, but it also increases the noise in the image and affects the image clarity.

[0062] 3. Image denoising - temporal denoising (also known as multi-frame denoising)

[0063] In a multi-frame scene, similar pixels can be found at similar pixel locations in different image frames. Therefore, by aligning, superimposing and averaging these pixels on multiple image frames, noise and noise in the image can be reduced, resulting in a clearer and smoother image. This image processing technique is often used to reduce image noise in low-light conditions and improve image quality.

[0064] 4. Image alignment technology

[0065] Image alignment technology is applied to image noise reduction of electronic devices in shooting scenarios, including feature point-based image alignment and hardware-based image alignment. Among them, hardware-based image alignment can be gyroscope-based image alignment.

[0066] 5. Optical Image Stabilizer (OIS)

[0067] OIS corrects "optical axis deviation" through the floating lens of the lens. The principle is that the gyroscope inside the lens detects tiny movements and then transmits the signal to the processor. The processor calculates the displacement that needs to be compensated, and then compensates for it through the compensation lens group according to the direction and displacement of the lens shake, thereby overcoming the image blur caused by camera vibration.

[0068] The anti-shake range of OIS usually depends on the specific camera or lens model. Different manufacturers and models may have different anti-shake ranges. For example, the anti-shake range of OIS can be an angle of 1 degree and a displacement of less than 300um.

[0069] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0070] When shooting in a dark environment, in order to ensure that the captured image is as clear as possible, the electronic device can perform image noise reduction based on image alignment technology, so that a clear image can be displayed in the preview interface.

[0071] See also Figure 1A , Figure 1A FIG. 1 is a flow chart of an image alignment process provided by an embodiment of the present application. Figure 1A As shown, the electronic device 100 responds to the user operation, and after starting the photo application, the electronic device can obtain the image collected by the camera in real time. Among them, the electronic device 100 includes an image analysis module 101, and the image analysis module 101 can be an EVA (Engine for Visual Analytics) module in an image signal processor (image signal processor, ISP). The EVA module is used to enhance the visual analysis and image processing capabilities of the electronic device 100, and specifically can provide the following functions: geometric correction, motion estimation, and calibration control. Exemplarily, a submodule under the EVA module in the ISP, namely globalalign, is used to control the alignment between frames to perform image denoising for the subsequent image. In this submodule, three image alignment modules are included, a module based on feature point alignment (also called image matching), a module based on gyroscope matching image alignment, and an automatic selection module. Among them, the automatic selection module switches between "image alignment based on feature point alignment" and "image alignment based on gyroscope matching" according to different actual conditions.

[0072] from Figure 1A It can be seen that in the automatic selection mode, if the image noise is small and the signal-to-noise ratio is high, it means that the feature points obtained after feature detection of the image (for example, features obtained after processing closed boundary areas, edges, contours, intersections and corners) have a high credibility, and the electronic device 100 can use an alignment method based on image matching. If the image noise is large and the signal-to-noise ratio is low, it means that the credibility of the feature points obtained after feature detection of the image is low, and the electronic device 100 can use an alignment method based on gyroscope matching.

[0073] Therefore, if Figure 1A As shown, when the confidence of the feature points on the n-th frame image 1012 and / or the n-1-th frame image 1011 is high, it means that there are enough credible feature points in the n-th frame image 1012 and / or the n-1-th frame image 1011, that is, the feature points on the n-th frame image 1012 can correspond one to one with the feature points on the n-1-th frame image 1011. Therefore, the electronic device 100 can perform pixel (block) alignment on the n-th frame image 1012 and the n-1-th frame image 1011 based on the alignment method of image matching, so as to achieve the purpose of image multi-frame fusion noise reduction in the subsequent process.

[0074] When the confidence of the feature points on the nth frame image 1012 and / or the n-1th frame image 1011 is low, it means that there are not enough credible feature points in the nth frame image 1012 and / or the n-1th frame image 1011, that is, the feature points on the nth frame image 1012 may not have a one-to-one correspondence with the feature points on the n-1th frame image 1011. Therefore, the electronic device 100 can use the gyroscope to align the images to superimpose the nth frame image 1012 and the n-1th frame image 1011, so as to achieve the purpose of multi-frame fusion noise reduction. That is, the nth frame image 1012 and the n-1th frame image 1011 are forced to be superimposed together. Among them, if the electronic device 100 does not shake, or the subject does not move, the image captured by the electronic device 100 is a static picture, so the pictures on the n-th frame image 1012 and the n-1-th frame image 1011 are not much different. The n-th frame image 1012 and the n-1-th frame image 1011 are directly aligned based on the gyroscope matching alignment method and forced to be superimposed together during subsequent fusion to obtain a fused image, so that the purpose of noise reduction can be achieved in subsequent processing.

[0075] However, if the electronic device 100 is shaking or the subject is moving, the alignment process based on the gyroscope may cause ghost images to appear on the fused image obtained after the superposition. Figure 1B , Figure 1B Schematic diagram of a ghost provided by an embodiment of the present application. Figure 1B It can be seen that the confidence of the feature points on the nth frame image 1012 and / or the n-1th frame image 1011 is low, and the image on the nth frame image 1012 captured by the electronic device 100 moves relative to the n-1th frame image 1011, so the image captured by the electronic device 100 is a dynamic image, so the images on the nth frame image 1012 and the n-1th frame image 1011 are quite different. If the electronic device 100 forcibly superimposes the nth frame image 1012 and the n-1th frame image 1011 based on the alignment method of gyroscope matching, an abnormal area may appear in the fused image obtained, and the abnormal area is called a ghost.

[0076] The nth frame image 1012 may be an image captured at the current time, and the n-1th frame image 1011 may be an image captured at a time before the current time, or an image stored in a cache.

[0077] In summary, in automatic selection mode, Figure 1AThe image alignment method shown uses the confidence of the feature points as a judgment basis. When there are enough reliable feature points in the image, the image alignment can be performed using an alignment method based on image matching; when there are not enough feature points in the image, the image alignment can be performed using an alignment method based on gyroscope matching. Therefore, in the image alignment process, the choice of alignment method is related to the confidence of the feature points, and has nothing to do with whether the electronic device 100 is shaking or the subject is moving. Therefore, when there are not enough feature points, even if the electronic device 100 is shaking or the subject is moving, the electronic device 100 will perform image overlay based on the gyroscope. Figure 1B It can be seen that in the process of image superposition based on the gyroscope alignment method, due to the shaking of the electronic device during the shooting process or the movement of the shooting object, the image content difference between multiple frames of images (for example, the nth frame image 1012 and the n-1th frame image 1011) is too large. Therefore, after superimposing multiple frames of images based on the gyroscope alignment method, non-existent abnormal areas may appear in the image obtained, and the abnormal areas are called ghosts. Ghosts will affect the imaging quality and reduce the user experience.

[0078] In view of this, the present application provides an image processing method. Based on the image processing method, in a dark environment, the electronic device can obtain the gyroscope jitter amount and the range of optical image stabilization (OIS) for comparison. If the gyroscope jitter amount is within the range of OIS, the captured image is equivalent to static. Using the original strategy (such as Figure 1A The image alignment shown in the figure: image alignment based on gyroscope matching) can ensure a good noise reduction effect. If the gyroscope jitter exceeds the OIS range, you can use the alignment based on image matching. The alignment based on image matching will be changed to single-frame output due to insufficient feature points. Because the dynamic clarity requirement is lower than that of static, it can avoid the ghosting introduced by the gyroscope and improve the clarity of the dark light preview stream.

[0079] The image processing method provided in the present application is not limited to application in dark light environments, but can also be applied to other shooting environments where image noise reduction is required, such as shooting environments where the confidence of other feature points is low.

[0080] First, the electronic device in the embodiment of the present application may be equipped with Or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, as well as cellular phones, personal digital assistants (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and / or smart city devices, etc.

[0081] Figure 2 The hardware structure of the electronic device of the embodiment of the present application is exemplified. Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0082] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0083] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0084] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0085] The processor 110 may also be provided with a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0086] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0087] In the embodiment of the present application, the processor 110 can be used to run the camera application, and the processor 110 is used to control each template in the camera application to execute the image processing method provided by the present application to display the following Figure 4A-4D The preview interface or the video recording interface described in the embodiment.

[0088] It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0089] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0090] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.

[0091] Non-volatile memory may include disk storage devices and flash memory.

[0092] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage unit potential level; universal flash storage (UFS), embedded multi media card (eMMC), etc. according to the storage specification.

[0093] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0094] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0095] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0096] In the embodiment of the present application, the computer program code corresponding to the image processing method may be stored in the NVM. After starting the camera application, the corresponding program code may be loaded into the RAM. The processor 110 may directly read the program code in the RAM to process the image captured by the image sensor, such as image alignment processing before noise reduction.

[0097] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0098] The display screen 194 is used to display images, videos, user interfaces, controls, windows, etc. In one implementation, the display screen 194 is used to display a smooth rounded corner obtained by a smooth rounded corner drawing method provided in the present application. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In one embodiment, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0099] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0100] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In one embodiment, ISP can be set in camera 193.

[0101] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0102] The camera 193 may include one or more of the following cameras: a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, a zoom camera, or a depth camera. Among them, the telephoto camera has a small shooting range and is suitable for shooting distant objects; the wide-angle camera has a larger shooting range; the ultra-wide-angle camera has a larger shooting range than the wide-angle camera and is suitable for shooting panoramic and other large-screen scenes. The depth camera can be used to measure the object distance of the object to be photographed, that is, to measure the depth information of the object to be photographed, for example, it may include a three-dimensional (3D) depth camera, a time of flight (TOF) depth camera, or a binocular depth camera.

[0103] The camera 193 may include a main camera and a secondary camera. The main camera may be used to capture images, for example, it may include a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a zoom camera, etc. The secondary camera may be used for ranging or other auxiliary functions, for example, it may include a depth camera.

[0104] The camera 193 may include a front camera and / or a rear camera. The front camera may include one or more main cameras, and the rear camera may also include one or more main cameras. When capturing an image, the target main camera used by the electronic device to capture the image may be a default main camera or a main camera selected by the user.

[0105] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0106] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0107] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or listen to a hands-free call through the speaker 170A.

[0108] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be received by placing the receiver close to the human ear.

[0109] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device can be provided with at least one microphone 170C. In other embodiments, the electronic device can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording function, etc.

[0110] The earphone interface 170D is used to connect a wired earphone and can be a USB interface, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0111] The electronic device 100 can determine the motion posture of the electronic device 100 through the gyro sensor 180B. In one embodiment, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used in scenes such as navigation and somatosensory games.

[0112] Exemplarily, in a low-light shooting environment, the gyroscope sensor 180B in the embodiment of the present application can be used to collect jitter information of the electronic device 100. If the jitter information is used to indicate that the jitter angle of the electronic device 100 during the shooting process is greater than the OIS anti-shake range, the electronic device 100 can process the image captured by the camera 193 based on the image matching alignment method. When there are not enough credible feature points in the captured image, the image is not subjected to multi-frame fusion, but is directly output.

[0113] The structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device may also include a pressure sensor 180A, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, an ambient light sensor 180L, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, a bone conduction sensor 180M, a button 190, a motor 191, and the like.

[0114] The software system of an electronic device (such as a mobile phone) can adopt a layered architecture, a transaction-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. This application embodiment takes the Android system with a layered architecture as an example to illustrate the software architecture of a mobile phone. Figure 3 , Figure 3 It is a schematic diagram of the software architecture of the electronic device provided in the embodiment of the present application.

[0115] like Figure 3 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer (application), the application framework layer (frame), the hardware abstraction layer (HAL), the driver layer, and the hardware layer. Among them:

[0116] The application layer (application) may include a series of program packages. For example, the application may include camera, gallery and other applications. Among them, the camera application may include but is not limited to: UI module, camera module, gallery module, etc. The UI module may be a cameraUI module, which is mainly responsible for the human-computer interaction of the camera application, such as controlling the display of the preview interface and the preview screen therein, receiving and responding to user operations occurring in the preview interface. The camera module provides camera functions, focus functions, etc. The gallery module can be used to store photos taken by the user in the file system or a specific database of the electronic device for retrieval by applications such as the gallery.

[0117] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. It mainly involves the camera framework, which can include camera access interfaces such as camera extension libraries and camera services, and plays a role in connecting the upper and lower parts. It can interact with camera applications through the application API and can also interact with HAL through the HAL interface definition language (HIDL). The application framework layer can also include a window manager. The camera application and the gallery application can display the photos taken to the user with the support of the window manager.

[0118] The hardware abstraction layer (HAL) is an interface layer located between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer and a decision module. Among them, the camera hardware abstraction layer may provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera) and more camera devices.

[0119] The decision module stores a variety of image processing algorithms. For example, in an embodiment of the present application, the decision module may include a brightness algorithm, a jitter algorithm, etc. The decision module may determine the alignment of the image in combination with the brightness algorithm, the jitter algorithm, and the shooting parameters reported by the camera module and / or other sensors.

[0120] In one implementation, the decision module can judge the image gain according to the brightness algorithm. If the image gain is greater than or equal to a preset threshold, it means that the shooting environment is a dark environment; if the image gain is less than the preset threshold, it means that the shooting environment is a bright environment.

[0121] Among them, image gain usually refers to the gain applied in image processing or image sensors to enhance the brightness or contrast of the image captured by the camera. The unit of gain is usually expressed in "multiples", such as 1x, 2x, 3x...16x, and these increased multiples are usually linear. Due to different usage habits of different manufacturers, "decibel (dB)" is sometimes used as a unit to measure gain. This gain can be applied at different stages of image signal processing, including at the sensor level, analog-to-digital converter level, or digital signal processing level. In image sensors, gain can be achieved by adjusting the gain setting of the sensor to enhance the brightness and contrast of the image. In digital image processing, gain can be achieved by adjusting the pixel value to enhance the brightness and contrast of the entire image. It is understandable that in normal environments, such as outdoor sunlight environments and indoor strong light environments, the gain is usually not increased to keep it at 1x. As the ambient brightness decreases, the gain needs to be slowly increased. When the environment is darker, the camera collects less light, and the camera will increase the exposure time and expand the aperture to increase the image brightness, so the gain needs to be increased, so the image gain is relatively large.

[0122] In one implementation, the decision module can judge the gyroscope jitter amount according to the jitter algorithm. If the gyroscope jitter amount is not within the OIS anti-shake range, for example, the gyroscope jitter amount is greater than or equal to the maximum anti-shake amount within the OIS anti-shake range, it means that the captured image is in a jittering state; if the gyroscope jitter amount is within the OIS anti-shake range, for example, the gyroscope jitter amount is less than the maximum anti-shake amount within the OIS anti-shake range, it means that the image is equivalent to being in a static state. Among them, the gyroscope jitter amount refers to the measurement of tiny vibrations or fluctuations generated by the built-in gyroscope of the electronic device during the measurement process. If the jitter amount is within the OIS anti-shake range, it means that the OIS module can compensate for the jitter amount by compensating the lens group; if the jitter amount exceeds the OIS anti-shake range, it means that the OIS module cannot compensate for the jitter amount by compensating the lens group.

[0123] The driver layer is a layer between hardware and software, which includes drivers for various hardware. The driver layer may include camera device drivers, digital signal processor drivers, and image processor drivers. Among them, the camera device driver is used to drive the image sensor of one or more cameras in the camera module to collect images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0124] The hardware layer may include a camera module, an image signal processor, a digital signal processor, an image processor, and a gyroscope sensor.

[0125] The camera module may include one or more image sensors of the camera (for example, image sensor 1, image sensor 2, etc.). Optionally, the camera module may also include a lens group (including one or more lenses), a photosensitive element, an OIS module, etc. Among them, the lens group and the photosensitive element can be used to cooperate with each other to realize the conversion between optical signals and electrical signals. The OIS module is used to provide the lens group and / or the photosensitive element with the ability to move within a plane. Among them, the translation capability within a plane is expressed as the movable distance on the X-axis and / or the Y-axis. Therefore, the OIS anti-shake range includes the moving distance on the X-axis and / or the Y-axis provided by the OIS module to the lens group and / or the photosensitive element.

[0126] The following is an example of the workflow of electronic device software and hardware, using the scenario of starting a camera application.

[0127] In the embodiment of the present application, when the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer, and the kernel layer processes the user operation into an original input event (including touch coordinates, timestamp of the touch operation, and other information), and identifies the control corresponding to the input event. Taking the touch operation as an example of a user operation acting on a camera application, the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera capability by calling the camera device in the camera hardware abstraction layer, such as camera device 1. The camera hardware abstraction layer sends the instruction to the camera device driver of the driver layer, and the camera device driver can start the sensor corresponding to the camera device, such as sensor 1, and then collect the image light signal through sensor 1, and transmit the image light signal to the image signal processor for preprocessing to obtain a preprocessed image.

[0128] In one implementation, a brightness algorithm and a jitter algorithm are stored in an image signal processor. The image signal processor can determine the image alignment method before noise reduction by combining the brightness algorithm, the jitter algorithm, and the shooting parameters reported by the camera module and / or other sensors, and perform alignment processing on the image light signal based on the determined alignment method, and then perform noise reduction processing on the image obtained by fusion of multiple frames after the alignment processing, so as to obtain a pre-processed image with better effect and without ghosting.

[0129] In one implementation, the image signal processor transmits the preprocessed image back to the hardware abstraction layer through the camera device driver, and the hardware abstraction layer can send the preprocessed image to the decision module. Based on the support of the image signal processor and the digital signal processor, the decision module in the hardware abstraction layer can determine the image alignment of the preprocessed image before noise reduction in combination with the brightness algorithm, the jitter algorithm, and the shooting parameters reported by the camera module and / or other sensors, and perform alignment processing on the preprocessed image based on the determined alignment method, and then perform noise reduction processing on the image obtained by fusion of multiple frames after the alignment processing to obtain an image with better effect and without ghosting.

[0130] The hardware abstraction layer returns the image data to the camera application through the camera interface. Subsequently, the camera application and the gallery application can display the ghost-free photos to the user with the support of the window manager.

[0131] Exemplarily, in a dark environment, if the gyroscope jitter amount collected by the gyroscope sensor is within the OIS anti-shake range, the decision module determines that the alignment scheme of the image is the first alignment scheme, and judges the confidence of the feature points on the image based on the first alignment scheme. When there are enough credible feature points on the image, the image is pixel (block) aligned based on image matching, so that subsequent modules can achieve the purpose of multi-frame image denoising; when there are not enough credible feature points on the image, the image is aligned based on gyroscope matching and the image is superimposed subsequently, so as to achieve the purpose of image denoising.

[0132] Exemplarily, in a dark environment, if the amount of jitter collected by the gyroscope sensor exceeds the OIS anti-shake range, the decision module determines that the alignment scheme of the image is an alignment scheme based on image matching. Therefore, when there are not enough reliable feature points on the image, the image is not aligned by multi-frame fusion, but the frame image is directly output, thereby achieving the purpose of eliminating ghost images.

[0133] Figure 4A-4D It is a user interface of a group of camera applications provided in an embodiment of the present application.

[0134] Figure 4A It is a system desktop 411 of an electronic device provided in an embodiment of the present application. Figure 4A As shown, the system desktop 411 may include a status bar 412, a page indicator 413, and a plurality of application icons.

[0135] Among them, the status bar 412 may include one or more signal strength indicators of mobile communication signals (also known as cellular signals), wireless high fidelity (wireless fidelity, Wi-Fi) signal strength indicators, battery status indicators, time indicators, etc.

[0136] The page indicator 413 may be used to indicate the positional relationship between the currently displayed page and other pages.

[0137] The multiple application icons may include a settings application icon, an application market application icon, a browser application icon, a gallery application icon, a camera application icon, a contact book application icon, a phone application icon, a message application icon, etc. Not limited to the above icons, the system desktop 411 may also include other application icons, which are not listed here one by one. The multiple application icons may be distributed on multiple pages. The page indicator 413 may be used to indicate which page the user is currently browsing among the multiple pages carrying multiple applications. The user can browse other pages by sliding left and right.

[0138] Understandably, Figure 4A The user interface described below merely illustrates a possible user interface style of an electronic device taking a mobile phone as an example and should not constitute a limitation on the embodiments of the present application.

[0139] like Figure 4A As shown, the electronic device 100 may receive a user operation of opening a camera application, such as clicking a desktop icon of the camera application. In response to the operation, the electronic device may display a Figure 4B The user interface shown.

[0140] Figure 4B The user interface of the shooting and display service provided in the embodiment of the present application is exemplarily shown, also called a preview interface.

[0141] like Figure 4B As shown, the preview interface may include a mode bar 421, a shooting control 422, a preview interface 423, a review control 424, and a quick function area 426. Among them:

[0142] The mode bar 421 may include multiple camera mode options, such as "night scene", "portrait", "photo", "video recording", etc. Different camera modes can provide users with camera services with different effects. Users can select any of the multiple camera modes to take photos according to different needs. For example, "photo" can be the default camera mode for taking photos. "Video recording" is used to record videos. The "night scene" mode is suitable for photo scenes with dim light, such as at night. The "portrait" mode is suitable for photo scenes where the subject of the photo is a person. The electronic device 100 can also provide more shooting modes, such as "large aperture", "movie", "professional", etc., which are not listed one by one here.

[0143] The electronic device 100 may receive a user operation for switching modes, such as sliding left / right in the mode bar 421, and change the currently used photo mode according to the operation. For example, when receiving a leftward sliding of the mode bar 421 (left sliding operation) and causing the float to stop at the "portrait" option, the electronic device may switch to the "portrait" mode. By default, the electronic device first uses the "photo" mode.

[0144] When the "photographing" mode is used, the shooting control 422 is used to trigger photographing; when the "recording" mode is used, the shooting control 422 can be switched to a recording control, and the recording control is used to trigger recording. The electronic device can receive whether there is a user operation on the shooting control 422, such as a click operation. After receiving the user operation on the shooting control 422, the electronic device can generate a shooting instruction. The electronic device can obtain the image reported by the camera corresponding to the timestamp according to the shooting instruction, and then save it as a photo.

[0145] The image frames captured by the camera are displayed in the preview interface 423. In the "photographing" mode, the preview interface 423 is a photo-taking preview interface, and in the "recording" mode, the preview interface 423 is a recording preview interface. Here, the preview interface 423 can display in real time the images processed by the image processing algorithms corresponding to different modes, so that the user can perceive the photographing effects corresponding to different photographing modes in real time.

[0146] The review control 424 can be used to browse thumbnails of the photos / videos that have been taken. When receiving a user operation acting on the review control 424, the electronic device can also display the wonderful photos corresponding to the thumbnails.

[0147] The quick function area 426 may include an HDR function option 426A, an AI function option 426B, a flash function option 426C, a color mode function option 426D, a setting function option 426E, etc. The HDR function option 426A can be used to trigger the electronic device to use the HDR algorithm to fuse the image when it is turned on. The AI ​​function option 426B can be used to trigger the electronic device to recognize the photo scene in the preview screen when it is turned on. The current AI function option 426B is in the off state. The flash function option 426C can be used to trigger the electronic device to turn on or off the flash. The color mode function option 426D can be used to trigger the electronic device to use a color filter to process the image collected by the camera. The setting function option 426E can be used to set the photo parameters of the electronic device (for example, image size, image storage format, etc.).

[0148] exist Figure 4BIn the preview interface 423 shown, the electronic device 100 can automatically detect the current shooting environment based on the image gain, where the image gain is a gain that can enhance the brightness and contrast of the image (for example, increasing the exposure time and expanding the aperture). Figure 4B The shooting environment shown is a bright light environment, so there is no need to increase the image brightness and contrast, so the image gain of the captured Nth frame image is less than the preset threshold value, and the electronic device 100 can align and fuse the Nth frame image and the N-1th frame image pixels (blocks) based on image matching, so the image obtained after alignment and fusion is displayed in the preview interface 423. Among them, the Nth frame image is the image captured at the current time, and the N-1th frame image is one or more frames of images captured at the time before the current time, or an image stored in the cache. It should be noted that for different "photographing" modes, the image gain in this mode may correspond to different preset thresholds, and this application does not impose any restrictions on this.

[0149] exist Figure 4C In the preview interface 434 shown, the electronic device 100 automatically detects that the shooting environment at this time is a dark light environment based on the image gain. It can be understood that in a dark light environment, the image brightness and contrast need to be increased, so the image gain of the collected N-th frame image is greater than or equal to the preset threshold, and the electronic device 100 obtains the gyroscope jitter amount collected by the gyroscope sensor. If the gyroscope jitter amount is within the OIS anti-shake range, it means that the shooting picture at this time is in a static state, and the electronic device 100 can perform image alignment on the N-th frame image based on the first alignment scheme. Exemplarily, the electronic device 100 determines the confidence of the feature points on the Nth frame image based on the first alignment scheme. When there are enough credible feature points on the Nth frame image, the Nth frame image and the N-1th frame image are aligned and fused in terms of pixels (blocks) based on image matching, so that the image obtained after the alignment and fusion is displayed in the preview interface 434; when there are not enough credible feature points on the Nth frame image, the Nth frame image and the N-1th frame image are superimposed based on gyroscope matching, so that the image obtained after superposition is displayed in the preview interface 434.

[0150] Combination Figure 4C or Figure 4D It can be seen that due to the shaking of the electronic device 100 or the movement of the subject, the subject in the captured image has moved from position 1 to position 2. Figure 4DIn the preview interface 425 shown, the shooting environment is a dark light environment, and the electronic device 100 determines that the gyroscope jitter is not within the OIS anti-shake range, indicating that the shooting picture is in a shaking state. Since the requirements for dynamic clarity are lower than those for static clarity, the electronic device 100 can align the Nth frame image based on image matching. Since there are not enough reliable feature points in the Nth frame image in a dark light environment, the electronic device 100 does not perform multi-frame fusion alignment processing on the Nth frame image, and directly outputs a single frame for subsequent noise reduction processing. The image displayed in the preview interface 425 is the image after the noise reduction processing of the Nth frame image.

[0151] It can be seen that when the electronic device 100 shakes or the subject moves, since the electronic device 100 does not superimpose the Nth frame image and the N-1th frame image based on gyroscope matching, but directly outputs the Nth frame image, it can avoid ghosting caused by image superposition, thereby improving imaging quality and user experience.

[0152] It should be noted that, not limited to the above preview scenario, the user can also display an image without ghosts in the video recording interface in the video recording scenario. Figure 4B-Figure 4C The left swipe operation of the mode bar 421 shown. In response to the above operation, the electronic device 100 can switch the "photographing" mode to the "video recording" mode. After entering the "video recording" mode, the electronic device 100 can switch the preview interface corresponding to the original "photographing" mode to the preview interface corresponding to the "video recording" mode. Specifically, the electronic device 100 can switch the original photo taking control of the "photographing" mode to the start video recording control of the "video recording" mode. The electronic device 100 receives a user operation acting on the start video recording control. In response to the above operation, the electronic device 100 can start recording a video and display the recorded image in the video recording interface. When recording, the electronic device 100 performs alignment processing on the captured image before noise reduction according to the image gain and the gyroscope jitter amount. The solution for displaying an image without ghosting in the video recording interface can refer to the relevant description under the above-mentioned photo taking scenario, which will not be repeated here.

[0153] See also Figure 5 , Figure 5 is a flowchart of an image processing method provided by an embodiment of the present application, which is applied to Figure 2 The hardware architecture shown and Figure 3 The software architecture shown includes but is not limited to the following steps:

[0154] Step S501: In response to a user operation, a camera application is started to obtain a first frame of image captured by a camera.

[0155] Specifically, an application program that can start a camera, such as a camera application, is running in the electronic device, and an instruction to collect image data can be generated according to business needs, so as to obtain an image (i.e., the first frame image) collected by the camera in real time. Exemplarily, the electronic device sends an instruction to start the camera capability by calling the camera device of the camera application abstraction layer. The camera application abstraction layer sends the instruction to the camera device driver of the driver layer, and the camera device driver can start the sensor corresponding to the camera device, and then collect the image light signal through the sensor, and convert the image light signal into an electrical signal to obtain the image data.

[0156] It is understandable that a rotating buffer of a fixed size may be preset in the electronic device, and the rotating buffer is used to store a set of raw images recently reported by the camera. Among them, the image obtained after the photosensitive element in the camera converts the captured light source signal into an electrical signal can be called a raw image, also known as a RAW image. Exemplarily, the buffer in the electronic device 100 can be used to cache the raw images reported by the camera, and after each frame of the raw image reported by the camera is received, the electronic device 100 can write it into the buffer. When the buffer is full, after receiving a new frame of the image, the earliest written frame of the raw image can be read from the buffer, and then the newly received image can be stored. Based on different uses, the electronic device 100 can divert the raw images stored in the buffer to obtain different image streams, such as photo streams, preview streams, video streams, etc.

[0157] In one implementation, the image processing method provided in an embodiment of the present application can be applied to any of the above-mentioned image streams. The first frame image acquired by the electronic device 100 can be directly derived from the RAW image reported by the camera, or it can be directly derived from the RAW image stored in the rotating buffer. The embodiment of the present application does not impose any restrictions on this.

[0158] Step S502: Determine output of the first frame of image according to the image gain of the first frame of image and the amount of jitter collected by the gyro sensor.

[0159] Among them, image gain is used to indicate the brightness added to the first frame of the image. It is understandable that in a dark environment, the camera of an electronic device receives less light. In order to ensure that the captured image has appropriate brightness, it is necessary to increase the image brightness by adjusting the exposure time and expanding the aperture, thereby increasing the gain, and the noise is also greater. In a bright light environment, the camera of an electronic device receives normal light, and the captured image already has appropriate brightness, so there is no need to increase the image brightness by increasing the gain. The gyroscope jitter is used to indicate the brightness of the tiny vibrations or fluctuations generated by the built-in gyroscope sensor of the electronic device during the measurement process.

[0160] In a possible implementation, a brightness algorithm and a jitter algorithm are stored in the HAL of the electronic device. After the camera of the hardware layer collects the first frame of image, it is sent to the image signal processor ISP. When the ISP determines that it needs to be subjected to noise reduction processing, it sends the first frame of image upward to the HAL. In the HAL, when it is determined that the shooting environment at this time is a dark light environment based on the image gain of the first frame of image based on the brightness algorithm, the gyroscope jitter amount collected by it is obtained from the gyroscope sensor of the hardware layer. Continuing in the HAL, when it is determined that the shooting picture at this time is in a jittering state based on the jitter algorithm according to the gyroscope jitter amount, it is determined that the alignment mode before the noise reduction processing of the first frame of image is a single frame output. Therefore, the HAL sends the alignment mode (i.e., single frame output) of the first frame of image to the ISP of the hardware layer, and the ISP performs alignment processing and / or other processing on the first frame of image based on the alignment mode, such as noise reduction processing, saturation optimization processing, exposure compensation processing, etc.

[0161] In a possible implementation, a brightness algorithm and a jitter algorithm are stored in the ISP of the electronic device. After the camera of the hardware layer collects the first frame of image, it sends it to the ISP. When the ISP determines that the shooting environment at this time is a dark light environment based on the image gain of the first frame of image based on the brightness algorithm, it obtains the gyroscope jitter amount collected from the gyroscope sensor of the hardware layer. When it is determined that the shooting picture at this time is in a jittering state based on the gyroscope jitter amount based on the jitter algorithm, it determines that the alignment method before the noise reduction processing of the first frame of image is a single frame output. Therefore, the ISP performs alignment processing and / or other processing on the first frame of image based on the alignment method, such as noise reduction processing, saturation optimization processing, exposure compensation processing, etc.

[0162] Exemplarily, when the electronic device determines that the image gain of the first frame image is greater than or equal to a preset threshold value (thershold) based on the brightness algorithm, it determines that the shooting environment is a dark light environment. It is understandable that the image gain of the first frame image is greater than or equal to the preset threshold value, indicating that the camera has less light input when capturing the first frame image, which further indicates that the shooting environment at this time is a dark light environment, and the amount of light input that can be provided is insufficient to ensure that the first frame image has appropriate brightness, and the required gain is relatively large. Among them, different device models and different "photographing" modes may correspond to different preset thresholds.

[0163] Exemplarily, when the electronic device determines based on the jitter algorithm that the gyroscope jitter amount is greater than or equal to the optical image stabilization OIS value, it determines that the captured image is in a jitter state. Among them, the OIS value is used to indicate the amount of displacement caused by vibration that the electronic device can compensate. It can be understood that if the gyroscope jitter amount is greater than or equal to the OIS value, it means that the gyroscope jitter amount is not within the anti-shake range of OIS, and the electronic device cannot compensate for the jitter amount by compensating the lens group, so the image captured by the electronic device will be in a jitter state.

[0164] It should be noted that due to jitter, the image content between the multiple frames captured by the camera may be too different. If the multiple frames are fused for alignment according to the existing solution, ghost images may appear on the fused image. Therefore, when the present application detects that the electronic device is jittering or the subject is moving in a dark environment, it does not perform alignment processing by multi-frame fusion, but directly outputs a single frame, thereby avoiding the introduction of ghost images due to multi-frame fusion.

[0165] In a possible implementation, when the electronic device performs alignment processing before noise reduction, when the shooting environment is determined to be a bright light environment based on the image gain of the first frame image based on a brightness algorithm, a second frame image is acquired. Furthermore, a dedicated buffer is also provided in the electronic device, and the dedicated buffer is used to store images after alignment processing by multi-frame fusion. The second frame image may be an image captured by the camera before the first frame image or an image stored in a cache, such as an image in a rotating buffer or a dedicated buffer. Next, the electronic device determines a third frame image based on the first frame image and the second frame image. Further, the electronic device may perform pixel (block) alignment and fusion of the first frame image and the second frame image based on image matching to obtain a fused third frame image. Alternatively, the electronic device may use a gyroscope to perform image alignment to superimpose the first frame image and the second frame image to obtain a superimposed third frame image.

[0166] Step S503: Display the first frame image in the preview interface of the camera application.

[0167] Specifically, in a dark environment, when the shooting picture of the electronic device is in a shaking state, after the HAL obtains the processed first frame image, it returns the first frame image to the camera framework. After the camera extension receives the first frame image from the HAL, it can pass the first frame image to the camera application to trigger the display of the first frame image in the preview interface.

[0168] In another possible implementation, in a bright light environment or in a dark light environment, when the shooting picture of the electronic device is in a static state, the electronic device displays a third frame image obtained by fusion or superposition of the first frame image and the second frame image in the preview interface of the camera application.

[0169] It is understandable that the preview image displayed in the preview interface may be continuously refreshed along with the preview stream.

[0170] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of determining an alignment method provided in an embodiment of the present application. Figure 6As shown, the electronic device obtains the first frame image. Before performing noise reduction processing on the frame image, it can first determine whether the image gain of the first frame image is greater than or equal to a preset threshold. If the image gain of the first frame image is greater than or equal to the preset threshold, it is then determined whether the gyroscope jitter is greater than or equal to the OIS value. If the gyroscope jitter is greater than or equal to the OIS value, the alignment mode for the first frame image is determined to be single-frame output, that is, the first frame image is determined to be output. For specific examples, see Figure 4D .

[0171] Continue to see Figure 6 If the image gain of the first frame image is less than the preset threshold, the alignment method for the first frame image is determined to be multi-frame fusion, that is, the image obtained by aligning the first frame image with the second frame image is output. For specific examples, see Figure 4B .

[0172] Continue to see Figure 6 If the image gain of the first frame image is greater than or equal to the preset threshold, and the gyroscope jitter is less than the OIS value, the alignment method for the first frame image is determined to be multi-frame fusion, that is, the image obtained by aligning the first frame image with the second frame image is output. For specific examples, see Figure 4C .

[0173] Among them, the comparison between the image gain and the preset threshold, and the comparison between the gyroscope jitter amount and the OIS value can be referred to step S502 or other related descriptions, which will not be repeated here.

[0174] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0175] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0176] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0177] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0178] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0179] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0180] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0181] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0182] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0183] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0184] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0185] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0186] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0187] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0188] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0189] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a camera and a gyroscope sensor, the method includes: In response to a user operation, a camera application is started to obtain a first frame of image captured by the camera; Determine outputting the first frame of image according to an image gain of the first frame of image and the amount of jitter collected by the gyroscope sensor, wherein the image gain is used to represent the brightness added to the first frame of image; The first frame image is displayed in a preview interface of the camera application.

2. The method according to claim 1, characterized in that The step of determining to output the first frame of image according to the image gain of the first frame of image and the jitter amount collected by the gyroscope sensor comprises: When determining that the shooting environment is a dark light environment according to the image gain of the first frame image based on a brightness algorithm, obtaining a gyroscope jitter amount collected by the gyroscope sensor; When it is determined based on the jitter algorithm according to the jitter amount of the gyroscope that the captured image is in a jitter state, it is determined to output the first frame image.

3. The method according to claim 2, characterized in that The step of determining that the shooting environment is a dark light environment according to the image gain of the first frame image based on a brightness algorithm includes: When it is determined based on the brightness algorithm that the image gain of the first frame image is greater than or equal to a preset threshold, it is determined that the shooting environment is a dark light environment.

4. The method according to claim 2 or 3, characterized in that: The step of determining that the captured image is in a shaking state according to the shaking amount of the gyroscope based on a shaking algorithm includes: When it is determined based on the jitter algorithm that the jitter amount of the gyroscope is greater than or equal to the optical image stabilization OIS value, it is determined that the captured image is in a jitter state, wherein the OIS value is used to indicate the displacement amount caused by vibration that can be compensated by the electronic device.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: When it is determined that the shooting environment is a bright light environment based on the brightness algorithm according to the image gain of the first frame of image, or when it is determined that the shooting picture is in a stationary state based on the jitter algorithm according to the jitter amount of the gyroscope, a second frame of image is acquired, where the second frame of image includes an image captured before the first frame of image or stored in a cache; Determine a third frame image according to the first frame image and the second frame image; The third frame image is outputted, and the third frame image is displayed in a preview interface of the camera application.

6. The method according to any one of claims 2 to 4, characterized in that: The electronic device comprises a hardware abstraction layer HAL, and the brightness algorithm and the dithering algorithm are stored in the HAL.

7. A chip system, characterized in that: The chip system is applied to electronic equipment, and the chip system includes an image signal processor ISP; The ISP is used to receive a first frame of image captured by a camera of the electronic device; The ISP is used to determine the output of the first frame image according to the image gain of the first frame image and the jitter amount collected by the gyroscope sensor, wherein the image gain is used to represent the brightness increased for the first frame image.

8. The chip system according to claim 7, characterized in that: The ISP is specifically used for: When determining that the shooting environment is a dark light environment according to the image gain of the first frame image based on a brightness algorithm, obtaining a gyroscope jitter amount collected by the gyroscope sensor; When it is determined based on the jitter algorithm according to the jitter amount of the gyroscope that the captured image is in a jitter state, it is determined to output the first frame image.

9. The chip system according to claim 8, characterized in that: The ISP is specifically used for: When it is determined based on the brightness algorithm that the image gain of the first frame image is greater than or equal to a preset threshold, it is determined that the shooting environment is a dark light environment.

10. The chip system according to claim 8 or 9, characterized in that: The ISP is specifically used for: When it is determined based on the jitter algorithm that the jitter amount of the gyroscope is greater than or equal to the optical image stabilization OIS value, it is determined that the captured image is in a jitter state, wherein the OIS value is used to indicate the displacement amount caused by vibration that can be compensated by the electronic device.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1-6.

12. A computer program product comprising instructions, characterized in that When the computer program product is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 6.

13. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 6.

Images