Image processing method and related device

Abstract

The embodiment of the invention provides an image processing method and a related device, and relates to the technical field of terminals. The method comprises the steps that in a scene of a multi-person group photo, when eye opening and closing algorithm processing is executed, one frame of synthesized image and other multiple frames of images with low definition can be used for processing, so that the processing speed of the eye opening and closing algorithm processing can be increased, the memory occupation of the electronic equipment is reduced, and the performance and execution efficiency of the electronic equipment are improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related devices. Background Art

[0002] Some applications in electronic devices can support the camera function to meet the user's camera needs. When taking photos, there may be a scene where multiple people take photos together.

[0003] However, in the scenario of taking a group photo with multiple people, it takes a long time to generate an image, occupies a large amount of electronic device memory, reduces the performance of the electronic device, and affects the user experience. Summary of the invention

[0004] The image processing method and related devices provided in the embodiments of the present application can use a synthesized frame of image and multiple frames of lower-definition images for processing when executing the eye-opening and eye-closing algorithm in a scene where multiple people are taking a group photo. This can speed up the processing of the eye-opening and eye-closing algorithm and reduce the memory usage of the electronic device, thereby improving the performance and execution efficiency of the electronic device.

[0005] In a first aspect, an image processing method provided by an embodiment of the present application comprises:

[0006] In response to an operation for triggering a photo, M frames of the first type of image are collected, where M is a positive integer greater than or equal to 1; the M frames of the first type of image are synthesized into a first image; the target object in the first image is processed into an open-eye state using N frames of the second type of image to obtain a second image, where N is a positive integer greater than or equal to 1; wherein the N frames of the second type of image include images for preview display, the size of the second type of image is smaller than the size of the first type of image, and the target object is an object that is partially or completely in a closed-eye state in the first image. The embodiment of the present application performs processing of an open-and-closed-eye algorithm based on an image with lower clarity, so that a closed-eye object in a synthesized frame of image can be in an open-eye state. In this way, the object in the image can be in an open-eye state as much as possible, and the processing speed of the open-and-closed-eye algorithm processing is accelerated, and the memory usage of the electronic device is reduced, thereby improving the performance and execution efficiency of the electronic device.

[0007] In a possible implementation, before responding to the operation for triggering a photo, it also includes: the image sensor of the electronic device uses the first mode to collect N1 frames of the second type of image, and stores the N1 frames of the second type of image in a zero-delay ZSL queue, wherein N1 is a positive integer greater than or equal to 1, and the ZSL queue is used to store the preview display image; collecting M frames of the first type of image, including: the image sensor switches from the first mode to the second mode, and collects M frames of the first type of image based on the second mode, wherein the pixel of the image obtained by the second mode is higher than the pixel of the image obtained by the first mode. Before executing the photo process, the electronic device can be in the image preview stage, and the image sensor uses the first mode to obtain the image, and can display a relatively low-definition image on the interface, which can save memory space and improve the performance of the electronic device. When taking a photo, the image sensor can switch from the first mode to the second mode. In this way, a higher-definition image can be obtained to improve the user experience.

[0008] In a possible implementation, after collecting M frames of the first type of image, it also includes: the image sensor switches from the second mode to the first mode again, collects N2 frames of the second type of image based on the first mode, and stores the N2 frames of the second type of image in the ZSL queue; wherein N2 is a positive integer greater than or equal to 1, and the N frames of the second type of image are images obtained from N1 frames of the second type of image and / or N2 frames of the second type of image. After executing the photo-taking process, the image sensor can switch from the second mode to the first mode. In this way, the image sensor can continue to collect images for preview display, and the current newer images can be saved in the ZSL queue, thereby not affecting the preview function of the interface and improving the user experience.

[0009] In a possible implementation, before collecting M frames of the first type of image, the method further includes: transmitting a frame output mode and a frame number to an image sensor, wherein the frame output mode includes a mode for obtaining a second type of image, and the frame number is M frames; collecting M frames of the first type of image, including: based on the frame output mode and the frame number, the image sensor collects M frames of the first type of image. The image sensor collects M frames of the first type of image based on the frame output mode and the frame number, and can reasonably generate images with higher pixels according to the needs of actual image synthesis. In this way, images that occupy a large amount of memory space will not be generated in large quantities, which reduces the memory usage, thereby improving the performance and execution efficiency of the electronic device.

[0010] In a possible implementation, the electronic device includes an image signal processor ISP and a first module. After collecting M frames of first-category images, the electronic device further includes: the ISP performs image processing on the M frames of first-category images, and the image processing includes one or more of the following: noise filtering on the M frames of first-category images, histogram statistics on the M frames of first-category images, and RGB channel statistics on the M frames of first-category images; the ISP transmits the processed M frames of first-category images to the first module, and the first module converts the processed M frames of first-category images into M frames of Bayer format images based on the remosaic algorithm; the M frames of first-category images are synthesized into a first image, including: synthesizing the first image based on the M frames of Bayer format images. Since the camera driver of the electronic device cannot directly recognize and process the original image data obtained using the Quadra mode, the remosaic algorithm can be used to convert the original image data into a Bayer format image. In this way, the image synthesis algorithm and the eye opening and closing algorithm can be executed on the image during the photo shooting process, so as to obtain a better quality image and improve the user experience.

[0011] In a possible implementation, after responding to the operation for triggering the photo-taking, the method further includes: creating a camera path, wherein the camera path creation includes one or more of the following: initializing an algorithm for implementing image synthesis, initializing an algorithm for processing the target object in the first image as an open-eye state. The electronic device can create a corresponding appropriate algorithm path according to different photo-taking requests. Thus, the algorithm and module can be reasonably initialized, the execution of redundant code can be reduced, and the operating efficiency of the electronic device can be improved.

[0012] In a possible implementation, the target object in the first image is processed into an open-eye state using N frames of the second type of image to obtain a second image, including: identifying the target object in the first image; performing facial recognition on the target object to obtain position information of the facial region in the target object; based on the position information of the facial region, obtaining image data of the corresponding position in the N frames of the second type of image in an open-eye state; based on the image data in an open-eye state, processing the target object in the first image to obtain the second image. In this way, after being processed by the eye-opening and eye-closing algorithm, the closed-eye state of the object in the image can be displayed as an open-eye state, thereby obtaining a better quality image that better meets user needs and improves user experience.

[0013] In one possible implementation, the first mode includes a Binning mode, and the second mode includes a Quadra mode. Since the Binning mode combines four pixels into one pixel, the size of the image obtained using the Binning mode is smaller, but the clarity is lower. However, the pixels of the image converted by the remosaic algorithm in the Quadra mode will not change, so the size of the image obtained using the Quadra mode is larger, but the clarity is higher. According to the characteristics of different frame output modes, the embodiments of the present application can select different frame output modes in appropriate scenarios, thereby meeting the needs of image display and improving the performance of electronic devices.

[0014] In a second aspect, an embodiment of the present application provides an image processing device, which may be an electronic device, or a chip or chip system in an electronic device. The device may include a processing unit. The processing unit is used to implement any method related to processing performed by the electronic device in the first aspect or any possible implementation of the first aspect. When the device is an electronic device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements the method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements the method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit in the chip (for example, a register, a cache, etc.), or a storage unit in the electronic device located outside the chip (for example, a read-only memory, a random access memory, etc.).

[0015] Exemplarily, the processing unit is used to collect M frames of the first type of image in response to an operation for triggering photo taking; it is also used to synthesize the M frames of the first type of image into a first image; and specifically, it is also used to use N frames of the second type of image to process the target object in the first image into an open-eye state to obtain a second image.

[0016] In one possible implementation, the processing unit is used to capture N1 frames of second-category images using a first mode and store the N1 frames of second-category images in a zero-delay ZSL queue, and is also used to switch from the first mode to the second mode and capture M frames of first-category images based on the second mode.

[0017] In a possible implementation, the processing unit is used to switch from the second mode to the first mode again, collect N2 frames of second-category images based on the first mode, and store the N2 frames of second-category images in the ZSL queue.

[0018] In a possible implementation, the processing unit is used to transmit a frame output mode and a frame number to the image sensor; and is also used to collect M frames of the first type of image based on the frame output mode and the frame number.

[0019] In a possible implementation, the processing unit is used to perform image processing on the M frames of the first type of images, and is also used to transfer the processed M frames of the first type of images to the first module, and is specifically used to synthesize the first image based on the M frames of Bayer format images.

[0020] In a possible implementation, the processing unit is used to create a camera path.

[0021] In one possible implementation, the processing unit is used to identify the target object in the first image; it is also used to perform facial recognition on the target object to obtain position information of the facial area in the target object; it is specifically used to obtain image data in an eyes-open state at a corresponding position in N frames of the second type of image based on the position information of the facial area; it is specifically used to process the target object in the first image based on the image data in the eyes-open state to obtain a second image.

[0022] In a possible implementation, the first mode includes a Binning mode, and the second mode includes a Quadra mode.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0024] In a fourth aspect, the present application provides a chip or a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0025] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0026] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0028] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of a preview interface of a camera application provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of image processing during camera preview and photo-taking provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of a process of image preview and image photography provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0037] 1. Frame output mode: The image sensor can support multiple frame output modes, such as Binning mode and Quadra mode. In some scenarios, the frame output mode can also be called the image output mode.

[0038] (1.1)Binning mode.

[0039] In Binning mode, after the image sensor collects the original pixel array, it can add the induced charges corresponding to four adjacent pixels in the original pixel array together as a pixel point, which can also be understood as concentrating the light of four pixels on one pixel point. The brightness of the image obtained in this way is relatively high, so Binning mode is suitable for use in darker scenes.

[0040] Since the Binning mode combines four pixels into one pixel, assuming that the size of the original pixel array is 50MP, the size of the image obtained by using the Binning mode is 12.5MP. Therefore, the size of the image obtained by using the Binning mode is smaller, but the clarity is lower.

[0041] (1.2)Quadra mode.

[0042] In Quadra mode, the image sensor can arrange each color in four pixel combinations to obtain a four-unit original pixel array as raw image data. However, the camera driver of the electronic device cannot directly recognize and process the raw image data obtained using the Quadra mode. In a possible implementation, the remosaic algorithm can be used to convert the raw image data into a Bayer format image.

[0043] It is understandable that the pixels of the image converted by the remosaic algorithm will not change. Assuming that the size of the original pixel array is 50MP, the size of the image converted by the remosaic algorithm is still 50MP. Therefore, the image obtained by using the Quadra mode is larger in size but has higher definition. In some scenarios, the Quadra mode can also be called the Non-Binning mode or the Quadra Bayer mode.

[0044] 2. Terminology

[0045] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0046] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0047] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0048] 3. Electronic equipment

[0049] The electronic device of the embodiment of the present application may also be a terminal device in any form. For example, the electronic device may include: a mobile phone, a tablet computer, a PDA, a laptop computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, an electronic device in a 5G network or a public land mobile communication network (public land mobile communication network) to be evolved in the future. Mobile network, PLMN) and the like, the embodiments of the present application are not limited to this.

[0050] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0051] In addition, in the embodiments of the present application, the electronic device may also be an electronic device in an Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0052] The electronic devices in the embodiments of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0053] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0054] For example, Figure 1 A schematic structural diagram of an electronic device is shown.

[0055] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0056] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented by hardware, software, or a combination of software and hardware.

[0057] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. For example, in an embodiment of the present application, the ISP can be used to process the image captured by the image sensor, for example, including color correction, white balance, noise reduction, color enhancement, etc.; it can also be used to parse and calculate the RAW image, for example, including noise filtering of the image, and histogram statistics, RGB channel statistics of the image, etc.; it can also be used to perform digital signal processing on the image, for example, including light compensation, lens distortion correction, exposure control, etc.

[0058] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the above-mentioned memory. Repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved. For example, in an embodiment of the present application, the processor 110 may be used to transmit instructions for capturing images, process images, execute multi-frame synthesis algorithms, etc.

[0059] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0060] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data created during the use of the electronic device, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor. For example, in an embodiment of the present application, the internal memory 121 can be used to store preview images, multi-frame synthesis algorithm-related codes, and eye opening and closing algorithm-related codes, etc.

[0061] The camera 193 is used to capture static images or videos. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1. For example, in the embodiment of the present application, the camera 193 can be used to capture images in scenes such as previewing, taking pictures, recording or video.

[0062] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1. The electronic device implements the display function through a GPU, a display screen 194, and an application processor. For example, in an embodiment of the present application, the display screen 194 can be used to display a preview image or a photo image, etc.

[0063] Figure 2 It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0064] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into the application layer, application framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer from top to bottom.

[0065] The application layer can also be called the application layer, which can include a series of application packages. Figure 2 As shown, the application package may include applications such as camera, gallery, video, etc. Applications may include system applications and third-party applications.

[0066] The application framework layer can also be called the Framework layer. The Framework layer can provide an application programming interface (API) and a programming framework for the application layer's applications. The Framework layer can include some predefined functions.

[0067] like Figure 2 As shown, the Framework layer may include a camera access interface. The camera application of the application layer may call the camera access interface to implement management of the camera device, etc.

[0068] The application layer and the framework layer run in a virtual machine. The virtual machine executes the java files of the application layer and the framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection. For example, in the embodiment of the present application, the virtual machine can be used to perform functions such as image processing, multi-frame synthesis algorithm, open and close eye algorithm, etc.

[0069] The hardware abstraction layer is an abstract structure between the kernel layer and the Android runtime. The hardware abstraction layer can be a package of hardware drivers, providing a unified interface for upper-layer applications to call.

[0070] The hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. In the embodiment of the present application, the camera hardware abstraction layer may include a Binning adapter and a Quadra adapter. The camera algorithm library includes a multi-frame synthesis algorithm and an eye opening and closing algorithm.

[0071] The Binning adapter can be used to pass ISP-processed images to the zero shutter lag (ZSL) queue.

[0072] The Quadra adapter can be used to pass the ISP processed image to the multi-frame synthesis algorithm and / or the open and closed eye algorithm. It is understood that the above-mentioned process of executing the remosaic algorithm can be implemented in hardware or in software. When implemented in software, the remosaic algorithm can be executed based on the Quadra adapter to achieve image format conversion.

[0073] The multi-frame synthesis algorithm can process multiple frames of images and synthesize them into one frame with higher definition and better quality.

[0074] The eye-opening and eye-closing algorithm can detect the face state in the image and determine whether the eyes are open or closed; it can also process the closed eye state as the open eye state based on multiple frames of images.

[0075] The driver layer is used to drive hardware resources. The driver layer can include multiple driver modules. The driver layer can include camera device drivers, DSP drivers, and GPU drivers.

[0076] The hardware layer includes image sensors, image signal processors (ISPs), digital signal processors (DSPs), and graphics processors (GPUs). The image sensor can generate frames based on the sensor's photo mode. The image signal processor (ISP) can process the images generated by the image sensor accordingly.

[0077] It should be noted that the embodiments of the present application are only illustrated using the Android system as an example. In other operating systems (such as Windows system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of the present application, the solutions of the present application can also be implemented.

[0078] Some applications in electronic devices can support the camera function to meet the user's photography needs.

[0079] When taking photos, there may be a scene of taking photos with multiple people. In the scene of taking photos with multiple people, users usually hope that the photos taken are clear, so that everyone's face can be seen and everyone has their eyes open.

[0080] In a possible implementation, in a group photo scene, in order to obtain a higher definition image, the electronic device can instruct the image sensor to use the Quadra mode to produce frames. The electronic device can pass multiple frames of higher definition images to a multi-frame synthesis algorithm, and the multi-frame synthesis algorithm can fuse the multiple frames into one frame. The electronic device can also pass the synthesized frame image together with other higher definition images to the eye opening and closing algorithm for processing, thereby obtaining a group photo.

[0081] However, in the above process, the eye opening and closing algorithm can use a smaller and lower definition image for eye opening and closing processing. However, the image size obtained by using the Quadra mode is larger, the eye opening and closing algorithm processing time is longer, and more electronic device memory is occupied, which reduces the performance of the electronic device and thus affects the execution efficiency of the electronic device.

[0082] In view of this, the image processing method provided in the embodiment of the present application can use a synthesized frame of image and other frames of lower-definition images for processing when executing the open and closed eyes algorithm processing in a scene where multiple people are taking a group photo. This can speed up the processing speed of the open and closed eyes algorithm processing and reduce the memory usage of the electronic device, thereby improving the performance and execution efficiency of the electronic device.

[0083] Figure 3The preview interface 301 of the camera application is shown. The interface 301 may also be referred to as the shooting interface of the camera application. The interface 301 may include a preview image 302, a zoom ratio, a shooting mode area, an album icon, a shooting control 303, and a camera rotation control.

[0084] The zoom ratio may be set to 1X by default, and the user may select a different zoom ratio by operating on the touch screen when using the shooting function.

[0085] The shooting mode area may include aperture, night scene, portrait, photo, video, short video, and more.

[0086] The album icon can display thumbnails of images. The user can click the album icon to view the photos or videos taken. The user can click the shooting control 303 to take a photo. The camera rotation control can be used to switch cameras.

[0087] Understandably, Figure 3 The photo-taking scene in the embodiment is only a schematic illustration of an application scenario of the present application, and does not limit the present application. The present application can also be applied to other scenarios using camera devices, such as video recording, video call, video live broadcast, etc.

[0088] Figure 4 A schematic diagram showing image processing during camera preview and photo-taking is shown.

[0089] S401, camera preview.

[0090] The user may trigger an operation to open the camera application. The operation to open the camera application may include clicking on the camera application icon, inputting a voice command to open the camera application, or triggering a shortcut key on the electronic device, etc. The specific operation to open the camera application is not limited in the embodiments of the present application.

[0091] In response to the user triggering the operation of opening the camera application, the electronic device can open the camera application and display the camera preview interface. Figure 3 As shown in the interface 301 , a preview image 302 may be displayed in the camera preview interface 301 .

[0092] The operation of triggering the opening of the camera application may include the operation of clicking the icon of the camera application, and may also include the operation of opening the camera application by voice instruction, etc. The specific operation method of triggering the opening of the camera application is not limited in the embodiment of the present application.

[0093] When the camera application displays the preview image, the image sensor can use the Binning mode for framing. It is understandable that in order to improve the performance of the electronic device, an image with appropriately smaller pixels can be used as the preview image during the camera preview process. Therefore, the Binning mode can be used for framing. The size of the image obtained in this way is smaller, and the memory of the electronic device is smaller, thereby improving the performance of the electronic device without affecting the user experience.

[0094] After being processed by the image signal processor, the captured image can be stored in the ZSL queue. The camera application can display a preview interface based on the image in the ZSL queue.

[0095] It is understandable that a certain number of images can be stored in the ZSL queue. As the image sensor continuously collects new images, the ZSL queue can delete the earlier stored images to store the newly generated images. In this way, the ZSL queue can store the current newer images.

[0096] S402: The camera application sends a photo-taking request.

[0097] The user may trigger a photo-taking operation, wherein the photo-taking operation may include clicking a photo-taking button, inputting a photo-taking instruction by voice, or triggering a shortcut key on an electronic device, etc. The specific photo-taking operation is not limited in the embodiments of the present application. Figure 3 In the interface 301, the user can click the shooting control 303 to trigger taking a photo.

[0098] In response to a user triggering a photo-taking operation, the camera application may send a photo-taking request to the camera hardware abstraction layer.

[0099] S403: The camera hardware abstraction layer creates a suitable path according to the photo request.

[0100] It is understandable that different photo-taking scenes may have different corresponding photo-taking requests, wherein the photo-taking scenes may include night scenes, architectural scenes, food scenes, group photo scenes, and the like.

[0101] Since the camera hardware abstraction layer may obtain different photo requests, the camera hardware abstraction layer may create corresponding appropriate algorithm paths according to different photo requests. The creation of algorithm paths may include initialization of various modules or algorithms under different photo scenes.

[0102] For example, the algorithm path can initialize the face detection algorithm, so that during the preview process, the face detection algorithm can be used to determine whether there is a face in the currently acquired image. If there is a face, the algorithm path can also initialize the beauty algorithm, and the beauty algorithm can be used to beautify the person in the image.

[0103] Taking the group photo mode as an example, creating an algorithm pathway may also include processes such as initializing a multi-frame synthesis algorithm and initializing an eye opening and closing algorithm.

[0104] S404: The camera hardware abstraction layer configures the frame mode and frame number requirements, and sends them to the image sensor.

[0105] It is understandable that in the previous processing method, both the multi-frame synthesis algorithm and the eye opening and closing algorithm use high-definition images for processing. For example, the multi-frame synthesis algorithm requires at least 5 frames of high-definition images, and the eye opening and closing algorithm requires at least 7 frames of high-definition images, so at least 12 frames of high-definition images are required. Therefore, the camera hardware abstraction layer needs to configure the frame mode to Quadra mode with 12 frames.

[0106] In the embodiment of the present application, since the eye opening and closing algorithm can be processed based on a frame of a higher-definition image synthesized and 7 lower-definition images, and the multi-frame synthesis algorithm requires at least 5 frames of higher-definition images, the camera hardware abstraction layer can configure the frame mode to be Quadra mode with 5 frames.

[0107] It is understandable that, assuming that the size of the higher definition image obtained by using the Quadra mode is 50MP, and the size of the lower definition image obtained by using the Binning mode is 12.5MP, then in the previous processing method, the image occupies at least 12×50MP in memory, that is, 600MP. In the embodiment of the present application, the image occupies at least 50MP+7×12.5MP in memory, that is, 137.5MP.

[0108] In summary, the image sensor can reasonably generate images with higher pixels based on the frame output mode and frame number to collect images according to the actual image synthesis needs. In this way, more images that occupy a large memory space will not be generated, which reduces the memory usage and improves the performance and execution efficiency of electronic devices.

[0109] It is understandable that the number of frames configured by the specific camera hardware abstraction layer can be set by the electronic device based on experience, and the embodiments of the present application do not limit it. For example, it is a more reasonable setting that the eye opening and closing algorithm requires at least 7 frames of images. This is because 7 frames can cover the process from the closed eye state to the open eye state.

[0110] In some implementations, the image sensor can collect 30 frames of images per second, so one frame of image corresponds to about 33 milliseconds (ms), and 7 frames of image can correspond to about 231ms. According to laboratory tests, the time it takes for a person to go from a closed eye state to an open eye state is also within the range of 231ms.

[0111] S405, the preview is paused and switched to quadra mode. After the frame number is met, the mode is switched back to Binning mode and the preview is resumed.

[0112] After the image sensor obtains the frame output mode and frame number, it can stop collecting preview images and switch to quadra mode to output frames. When the frame number meets the requirements, it can switch back to Binning mode and continue to collect preview images.

[0113] S406. ISP sends the frame to a multi-frame synthesis algorithm.

[0114] The image sensor can pass the collected high-definition images to the ISP. After the image processing by the ISP, the ISP can pass the image to the multi-frame synthesis algorithm through the quadra adapter. Then the multi-frame synthesis algorithm can synthesize the multi-frame images.

[0115] S407 , the eye opening and closing algorithm obtains the cached image from the ZSL queue.

[0116] The eye opening and closing algorithm can obtain images with lower definition from the ZSL queue for processing. The images in the ZSL queue may include images saved in the ZSL queue before switching the quadra mode, and may also include images re-captured and saved in the ZSL queue by the image sensor after switching the quadra mode, which is not limited in the embodiments of the present application.

[0117] In this way, on the one hand, the eye opening and closing algorithm obtains the cached image from the ZSL queue, which can reduce the time for the image sensor to re-produce frames; on the other hand, the image cached in the ZSL queue is an image obtained based on the Binning mode, and the image size is relatively small, which saves memory usage of electronic devices.

[0118] S408, the eye opening and closing algorithm returns a result.

[0119] After being processed by the multi-frame synthesis algorithm and the eye opening and closing algorithm, a group photo with high definition and the characters in the image with open eyes can be obtained and displayed to the user through the camera application.

[0120] Figure 5 A schematic diagram of the process of image preview and image capture is shown.

[0121] The image preview process may include steps S501 to S507, and the image capture process may include steps S508 to S518.

[0122] S501: Receive an operation to trigger opening a camera. A user may trigger an operation to open a camera application, and in response to the operation, the electronic device may open the camera application at the application layer.

[0123] For specific operations to open the camera application, please refer to the above Figure 4 The relevant description in step S401 of the corresponding embodiment is not repeated here.

[0124] When the camera application displays the preview image, step S502 may be executed.

[0125] S502: The camera application transmits an instruction for capturing an image.

[0126] The camera application can pass instructions to the image sensor to capture an image.

[0127] Exemplarily, the camera application may call the camera access interface of the application framework layer to transmit an instruction to capture an image. It is understandable that the image sensor capturing an image may be referred to as sensor exposure.

[0128] The camera access interface can pass the instruction to the camera hardware abstraction layer, and the camera hardware abstraction layer can pass the instruction to the camera device driver of the driver layer. After obtaining the instruction, the camera device driver can instruct the image sensor of the hardware layer to start running, perform image exposure, and generate a RAW image.

[0129] After the image sensor obtains the instruction of the camera device driver, step S503 may be executed.

[0130] S503: The image sensor outputs frames in Binning mode to generate a RAW image.

[0131] It is understandable that in the image preview interface, the image sensor can use the Binning mode to output frames. Since the pixels of the image obtained by using the Binning mode are relatively low, this can reduce the memory usage of the electronic device, improve the performance of the electronic device, and thus reduce interface freezes.

[0132] After the image sensor generates a RAW image, step S504 may be further performed.

[0133] S504: The image sensor transmits the RAW image to the ISP.

[0134] S505: ISP processes the RAW image.

[0135] After acquiring the RAW image, the ISP can perform analysis and calculation on the RAW image. For example, the ISP can filter the noise of the image, and the ISP can also perform histogram statistics and RGB channel statistics on the image to obtain the processed image information.

[0136] After the ISP processes the RAW image, step S506 may be performed.

[0137] S506: ISP returns processed image information.

[0138] The ISP can return processed image information to the camera application.

[0139] Exemplarily, the ISP may pass processed image information to the camera device driver, the camera device driver may pass the processed image information to the Binning adapter of the camera hardware abstraction layer, and then the Binning adapter may pass the image to the ZSL queue for storage.

[0140] The camera application may call a related camera access interface to obtain an image from the ZSL queue and execute step S507.

[0141] S507: The camera application displays a preview image.

[0142] S508: Receive an operation to trigger taking a photo.

[0143] After the camera application receives the user's operation of triggering a photo-taking, it may start taking photos in response to the operation and execute step S509.

[0144] For specific photo-taking operations, please refer to the above Figure 4 The relevant description in step S402 of the corresponding embodiment is not repeated here.

[0145] S509: Transmit a photo-taking instruction.

[0146] Similar to the above step S502, the camera application may transmit a photographing instruction to the image sensor, wherein the photographing instruction may include a frame output mode and a frame number, and may also include parameters such as an exposure time.

[0147] For example, the camera application may call the camera access interface to pass the photo command. The camera access interface may pass the photo command to the camera hardware abstraction layer, and the camera hardware abstraction layer may pass the photo command to the camera device driver. After obtaining the photo command, the camera device driver may instruct the image sensor to perform image exposure and generate a RAW image.

[0148] After the image sensor obtains the instruction of the camera device driver, step S510 may be executed.

[0149] S510, using Quadra mode to generate frames and generate RAW images.

[0150] If the currently captured image is a group photo of multiple people, the image sensor can use Quadra mode to output frames and generate a RAW image.

[0151] In a possible implementation of determining a group photo scene, during the above image preview process, the camera application can detect faces in the image, and when the number of detected faces exceeds a preset value, it is determined that the current scene is a group photo. The preset value can be set by the electronic device based on experience, for example, the preset value can be 3 or 5, etc. The specific value of the preset value is not limited in the embodiment of the present application.

[0152] In another possible implementation of determining a group photo scene, the camera application may also provide a group photo shooting mode. Figure 3 In the interface 301 of the camera application, the shooting mode area may also include a control corresponding to the group photo mode (not shown in the figure), or other interfaces of the camera application, such as more interfaces of the camera application, may include a control corresponding to the group photo mode (not shown in the figure). The group photo mode may be in any possible position in the camera application, and the embodiment of the present application is not limited thereto.

[0153] When the user triggers the operation of the control corresponding to the group photo mode, in response to the user's triggering operation, the camera application can determine that the current scene is a group photo, and further instruct the image sensor to use the Quadra mode to output frames.

[0154] S511: The image sensor transmits the RAW image to the ISP.

[0155] S512: ISP processes the RAW image.

[0156] After the ISP processes the RAW image, step S513 may be performed. The specific ISP processing of the RAW image may refer to the relevant description in the above step S505, which will not be repeated here.

[0157] S513. The ISP returns the processed image information to the Quadra adapter.

[0158] S514. The Quadra adapter returns the remosaic algorithm result.

[0159] The Quadra adapter can execute the remosaic algorithm, convert the image generated by the Quadra mode into the Bayer format, and return the image after executing the remosaic algorithm to the multi-frame synthesis algorithm.

[0160] S515. The multi-frame synthesis algorithm returns the image synthesis result.

[0161] The multi-frame synthesis algorithm can fuse multiple frames of images into one frame of image and return the result of image synthesis to the eye opening and closing algorithm.

[0162] S516. The ISP returns the processed image information to the Binning adapter.

[0163] After the image sensor generates frames based on the Quadra mode and the frame number, the image sensor can generate frames again using the Binning mode and transmit the RAW image to the ISP. After the ISP processes the image captured by the image sensor, it can return the processed image information to the camera application.

[0164] Exemplarily, the ISP may transmit the processed image information to the camera device driver, the camera device driver may transmit the processed image information to the Binning adapter, and the Binning adapter may transmit the image to the ZSL queue for storage.

[0165] The camera application can call the relevant camera access interface to obtain images from the ZSL queue.

[0166] S517, the eye opening and closing algorithm returns the eye opening and closing algorithm result.

[0167] On the one hand, the eye opening and closing algorithm can obtain the synthesized one-frame image returned by the multi-frame synthesis algorithm in step S515. On the other hand, the eye opening and closing algorithm can obtain multiple frames of images with lower definition from the ZSL queue.

[0168] The eye-opening and eye-closing algorithm processes the closed-eye object in a synthesized frame of image as an open-eye state based on the above-mentioned multiple frames of images with lower definition, and returns the processed image to the camera application through the camera access interface.

[0169] In a possible implementation, the eye-opening and eye-closing algorithm can identify whether the image includes a closed-eye object based on the eye-opening and eye-closing detection in face recognition, or it can classify eye images based on a pre-trained neural network model to identify the closed-eye state.

[0170] The eye-opening and eye-closing algorithm can also process the eye-closed object in the image based on the image data of the eye-open object, thereby processing the eye-closed object into an eye-open state.

[0171] Exemplarily, the eye-opening and eye-closing algorithm can identify the closed-eye object in the composite image, and perform facial recognition on the closed-eye object, thereby obtaining the position information of the facial region in the closed-eye object. The eye-opening and eye-closing algorithm can also obtain image data of the open-eye state at the corresponding position in multiple frames of low-definition images based on the position information of the facial region, and process the closed-eye object in the composite image based on the image data to obtain an image including the object in the open-eye state.

[0172] S518: The camera application displays the photographed image.

[0173] After being processed by the multi-frame synthesis algorithm and the eye opening and closing algorithm, a group photo with high definition and the characters in the image with open eyes can be obtained and displayed to the user through the camera application.

[0174] It is understandable that the Tarce tool can be used to obtain the time consumption and memory usage of the photo process. In a group photo scene, multiple groups of frame times can be obtained by taking multiple photos. Compared with the current frame method, the image processing method of the embodiment of the present application can use less frame time and occupy less memory space.

[0175] The following specific embodiments are used to describe the method of the present application in detail. The following embodiments may be combined with each other or implemented independently, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0176] Figure 6 The image processing method of the embodiment of the present application is shown. The method includes:

[0177] S601: In response to an operation for triggering photo taking, M frames of first-category images are collected, where M is a positive integer greater than or equal to 1.

[0178] In the embodiment of the present application, the operation of triggering the photo shooting can refer to the above Figure 4 The operation of taking photos in step S402 of the corresponding embodiment will not be described in detail.

[0179] The first type of image can be understood as an image with higher pixel, higher definition, and larger memory usage. For example, the first type of image can include an image obtained by using the Quadra mode in the above embodiment.

[0180] S602: Synthesize M frames of first-category images into a first image.

[0181] In the embodiment of the present application, the first image can be understood as an image with high quality and good definition based on the synthesis of multiple frames of images. The first image can be an image obtained based on the multi-frame synthesis algorithm in the above embodiment.

[0182] S603. Process the target object in the first image into an open-eye state using N frames of second-category images to obtain a second image, where N is a positive integer greater than or equal to 1; wherein the N frames of second-category images include images for preview display, the size of the second-category images is smaller than the size of the first-category images, and the target object is an object in the first image that is partially or completely in a closed-eye state.

[0183] In the embodiment of the present application, the second type of image can be understood as an image with lower pixels, lower definition, and smaller memory usage. For example, the first type of image can include images in the ZSL queue, and can also include images obtained using the Binning mode in the above embodiment.

[0184] The second image can be understood as an image obtained after being processed by the eye opening and closing algorithm in the above embodiment.

[0185] The embodiments of the present application can execute the processing of the eye-opening and eye-closing algorithm based on an image with lower definition, and specifically can include using a frame of image synthesized and other multiple frames of image with lower definition for processing, so that the closed-eye object in the synthesized frame of image can be in an eye-open state. In this way, the object in the image can be in an eye-open state as much as possible, and the processing speed of the eye-opening and eye-closing algorithm processing is accelerated, and the memory usage of the electronic device is reduced, thereby improving the performance and execution efficiency of the electronic device.

[0186] Optional, in Figure 6 On the basis of the corresponding embodiment, before responding to the operation for triggering photo taking, it may also include: the image sensor of the electronic device adopts the first mode to capture N1 frames of the second type of images, and stores the N1 frames of the second type of images in a zero-delay ZSL queue, wherein N1 is a positive integer greater than or equal to 1, and the ZSL queue is used to store the preview displayed images; capturing M frames of the first type of images, including: the image sensor switches from the first mode to the second mode, and captures M frames of the first type of images based on the second mode, wherein the pixels of the image obtained using the second mode are higher than the pixels of the image obtained using the first mode.

[0187] In the embodiment of the present application, the first frame output mode can be used to obtain an image with lower pixels, for example, the image with lower pixels can be used as a preview image. The second frame output mode can be used to obtain an image with higher pixels.

[0188] The process of switching the image sensor from the first mode to the second mode can refer to the above Figure 4 The relevant description in step S405 of the corresponding embodiment is not repeated here.

[0189] ZSL queue can refer to the above Figure 4 The relevant description in step S401 of the corresponding embodiment is not repeated here.

[0190] Before executing the photo process, the electronic device can be in the image preview stage, and the image sensor uses the first mode to obtain the image, and can display a relatively low-definition image on the interface, which can save memory space and improve the performance of the electronic device. When taking a photo, the image sensor can switch from the first mode to the second mode. In this way, a higher-definition image can be obtained, which improves the user experience.

[0191] Optional, in Figure 6On the basis of the corresponding embodiment, after acquiring M frames of first-category images, it may also include: the image sensor switches from the second mode to the first mode again, acquires N2 frames of second-category images based on the first mode, and stores the N2 frames of second-category images in the ZSL queue; wherein N2 is a positive integer greater than or equal to 1, and the N frames of second-category images are images obtained from N1 frames of second-category images and / or N2 frames of second-category images.

[0192] In the embodiment of the present application, the process of switching the image sensor from the second mode to the first mode can refer to the above Figure 4 The relevant description in step S405 of the corresponding embodiment is not repeated here.

[0193] After the photo-taking process is completed, the image sensor can switch from the second mode to the first mode. In this way, the image sensor can continue to collect images for preview display, and the ZSL queue can store the current newer images, thereby not affecting the preview function of the interface and improving the user experience.

[0194] Optional, in Figure 6 On the basis of the corresponding embodiment, before collecting M frames of the first category of images, it can also include: transmitting a frame output mode and a frame number to the image sensor, the frame output mode includes a mode for obtaining a second category of images, and the frame number is M frames; collecting M frames of the first category of images includes: based on the frame output mode and the frame number, the image sensor collects M frames of the first category of images.

[0195] In the embodiment of the present application, the frame output mode and the frame number can refer to the above Figure 4 The relevant description in step S404 of the corresponding embodiment is not repeated here.

[0196] The image sensor collects M frames of the first type of images based on the frame output mode and the number of frames, and can reasonably generate images with higher pixels according to the needs of actual image synthesis. In this way, more images that occupy a large memory space will not be generated, which reduces the memory usage, thereby improving the performance and execution efficiency of the electronic device.

[0197] Optional, in Figure 6 On the basis of the corresponding embodiments, the electronic device includes an image signal processor ISP and a first module. After collecting M frames of first-category images, it can also include: the ISP performs image processing on the M frames of first-category images, and the image processing includes one or more of the following: noise filtering on the M frames of first-category images, histogram statistics on the M frames of first-category images, and RGB channel statistics on the M frames of first-category images; the ISP transmits the processed M frames of first-category images to the first module, and the first module converts the processed M frames of first-category images into M frames of Bayer format images based on the remosaic algorithm; synthesizing the M frames of first-category images into the first image can include: synthesizing the first image based on the M frames of Bayer format images.

[0198] In the embodiment of the present application, the first module can be understood as a module that can convert the first type of image into a Bayer format image based on the remosaic algorithm. For example, the first module may include the Quadra adapter in the above embodiment. The naming and implementation of the specific first module are not limited in the embodiment of the present application.

[0199] For the specific implementation and interaction process of ISP and the first module, please refer to the above Figure 5 The relevant descriptions in the corresponding embodiments are not repeated here.

[0200] Since the camera driver of the electronic device cannot directly recognize and process the raw image data obtained using the Quadra mode, the remosaic algorithm can be used to convert the raw image data into an image in the Bayer format. In this way, the image synthesis algorithm and the eye opening and closing algorithm can be executed on the image during the photo shooting process, thereby obtaining a better quality image and improving the user experience.

[0201] Optional, in Figure 6 On the basis of the corresponding embodiment, after responding to the operation for triggering photo taking, it may also include: creating a camera path, and creating the camera path includes one or more of the following: initializing an algorithm for realizing image synthesis, and initializing an algorithm for processing the target object in the first image as an open-eye state.

[0202] In the embodiment of the present application, the process of creating a camera channel can refer to the above Figure 4 The relevant description in step S403 of the corresponding embodiment is not repeated here.

[0203] Electronic devices can create appropriate algorithm paths according to different photo-taking requests, thereby reasonably initializing algorithms and modules, reducing the execution of redundant codes, and improving the operating efficiency of electronic devices.

[0204] Optional, in Figure 6 On the basis of the corresponding embodiments, using N frames of the second type of images to process the target object in the first image into an open-eye state to obtain a second image can include: identifying the target object in the first image; performing facial recognition on the target object to obtain position information of the facial area in the target object; based on the position information of the facial area, acquiring image data in the open-eye state at the corresponding position in the N frames of the second type of images; based on the image data in the open-eye state, processing the target object in the first image to obtain the second image.

[0205] In the embodiment of the present application, the eye opening and closing algorithm can refer to the above Figure 5 The relevant description in step S517 of the corresponding embodiment is not repeated here.

[0206] After being processed by the eye-opening and eye-closing algorithm, the closed-eye state of the object in the image can be displayed as the open-eye state, thereby obtaining a better quality image that better meets user needs and improves the user experience.

[0207] Optional, in Figure 6 On the basis of the corresponding embodiment, the first mode includes the Binning mode, and the second mode includes the Quadra mode.

[0208] In the embodiments of the present application, the Binning mode may refer to the relevant description of the Binning mode in the above embodiments (1.1), and the Quadra mode may refer to the relevant description of the Quadra mode in the above embodiments (1.2), which will not be repeated here.

[0209] Since the Binning mode combines four pixels into one pixel, the size of the image obtained using the Binning mode is smaller, but the clarity is lower. The pixels of the image converted by the remosaic algorithm in the Quadra mode will not change, so the size of the image obtained using the Quadra mode is larger, but the clarity is higher. According to the characteristics of different frame output modes, the embodiments of the present application can select different frame output modes in appropriate scenarios, thereby meeting the needs of image display and improving the performance of electronic devices.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0211] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0212] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0213] like Figure 7 FIG. 7 is a schematic diagram of a chip structure provided by an embodiment of the present application. The chip 700 includes one or more (including two) processors 701 , a communication line 702 , a communication interface 703 and a memory 704 .

[0214] In some implementations, the memory 704 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0215] The method described in the above embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 701. The above processor 701 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 701 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0216] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 704, and the processor 701 reads the information in the memory 704 and completes the steps of the above method in combination with its hardware.

[0217] The processor 701 , the memory 704 , and the communication interface 703 may communicate with each other via the communication line 702 .

[0218] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0219] The embodiment of the present application also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium may include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0220] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0221] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0222] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device, and the method comprises: In response to an operation for triggering photographing, M frames of first-category images are acquired, where M is a positive integer greater than or equal to 1; Combining the M frames of first-category images into a first image; Using N frames of the second type of images, the target object in the first image is processed into an open-eye state to obtain a second image, where N is a positive integer greater than or equal to 1; The N frames of the second type of images include images for preview display, the size of the second type of images is smaller than the size of the first type of images, and the target object is an object in the first image that is partially or completely in a closed-eye state.

2. The method according to claim 1, characterized in that Before responding to the operation for triggering taking a photo, the method further includes: The image sensor of the electronic device collects N1 frames of the second type of images in the first mode, and stores the N1 frames of the second type of images in a zero-delay ZSL queue, wherein N1 is a positive integer greater than or equal to 1, and the ZSL queue is used to store images for preview display; The collecting of M frames of first-category images includes: The image sensor switches from the first mode to the second mode, and collects the M frames of the first type of images based on the second mode, wherein pixels of the image obtained by using the second mode are higher than pixels of the image obtained by using the first mode.

3. The method according to claim 2, characterized in that After acquiring M frames of the first type of images, the method further includes: The image sensor switches from the second mode to the first mode again, collects N2 frames of the second type of images based on the first mode, and stores the N2 frames of the second type of images in the ZSL queue; Wherein, the N2 is a positive integer greater than or equal to 1, and the N frames of the second category images are images obtained from the N1 frames of the second category images and / or the N2 frames of the second category images.

4. The method according to any one of claims 1 to 3, characterized in that: Before collecting M frames of first-category images, the method further includes: Transmitting a frame output mode and a frame number to an image sensor, wherein the frame output mode includes a mode for obtaining the second type of image, and the frame number is M frames; Collect M frames of the first type of images, including: Based on the frame output mode and the frame number, the image sensor collects the M frames of the first type of images.

5. The method according to any one of claims 1 to 4, characterized in that: The electronic device includes an image signal processor ISP and a first module. After collecting M frames of the first type of images, the electronic device further includes: The ISP performs image processing on the M frames of the first category of images, and the image processing includes one or more of the following: performing noise filtering on the M frames of the first category of images, performing histogram statistics on the M frames of the first category of images, and performing RGB channel statistics on the M frames of the first category of images; The ISP transmits the processed M frames of the first type of images to the first module, and the first module converts the processed M frames of the first type of images into M frames of Bayer format images based on a remosaic algorithm; Combining the M frames of first-category images into a first image, comprising: The first image is synthesized based on the M frames of Bayer format images.

6. The method according to any one of claims 1 to 5, characterized in that: After responding to the operation for triggering taking a photo, the method further includes: A camera path is created, wherein the camera path creation includes one or more of the following: initializing an algorithm for implementing image synthesis, initializing an algorithm for processing the target object in the first image as an open-eye state.

7. The method according to any one of claims 1 to 6, characterized in that: The method of processing the target object in the first image into an open-eye state by using N frames of the second type of images to obtain a second image includes: identifying the target object in the first image; Performing facial recognition on the target object to obtain position information of the facial area in the target object; Based on the position information of the facial region, acquiring image data of the corresponding position in the N frames of the second type of images in an eye-open state; Based on the image data in the eyes-open state, the target object in the first image is processed to obtain the second image.

8. The method according to any one of claims 2 to 7, characterized in that: The first mode includes a Binning mode, and the second mode includes a Quadra mode.

9. An electronic device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 8.

10. A chip system, characterized in that: The method comprises at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the computer executes the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables an electronic device to execute the method according to any one of claims 1 to 8.

Images