Image processing method and related apparatus

Abstract

Embodiments of the present application provide an image processing method and related apparatus, and relate to the technical field of terminals. The method comprises: in response to a photographing operation, obtaining N frames of images, N being an integer greater than 1; obtaining an image sequence after sorting the N frames of images; wherein the N frames of images in the image sequence are sorted according to fusion weights of the N frames of images when image fusion is performed, and the sorting of the N frames of images in the image sequence is related to a first sorting result and a second sorting result, the first sorting result being obtained by sorting the N frames of images using a first algorithm for blur processing, and the second sorting result being obtained by sorting the N frames of images using a second algorithm for open-eye detection; and performing image fusion based on the image sequence. In this way, the probability of closed eyes appearing in the image can be reduced.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] Electronic devices can support portrait photography. Currently, in portrait photography scenarios, when there are many people to be photographed, the images captured by electronic devices tend to show a large number of people with closed eyes, which cannot meet user needs. Summary of the Invention

[0003] This application provides an image processing method and related apparatus, which are applied in the field of terminal technology and help reduce the probability of images showing people with closed eyes.

[0004] In a first aspect, embodiments of this application propose an image processing method, comprising: in response to a photo-taking operation, acquiring N frames of images, where N is an integer greater than 1; obtaining an image sequence after sorting the N frames of images; wherein the N frames of images in the image sequence are sorted according to the fusion weights of the N frames of images during image fusion, and the sorting of the N frames of images in the image sequence is related to a first sorting result and a second sorting result, wherein the first sorting result is obtained by sorting the N frames of images using a first algorithm for jitter processing, and the second sorting result is obtained by sorting the N frames of images using a second algorithm for eye-opening / closing detection; and performing image fusion based on the image sequence. This helps to reduce the probability of closed eyes appearing in the image.

[0005] In one possible implementation, the second sorting result is obtained as follows: N frames of images are cropped to obtain N cropped frames, which include the eye region; a second algorithm is used to identify the cropped N frames to obtain the open / closed eye results, which represent the number of open and / or closed eyes in the image, or the probability of open eyes; based on the open / closed eye results, the N frames are sorted to obtain the second sorting result. Thus, since the resolution of the cropped N frames is lower than that of the original N frames, cropping the N frames before performing open / closed eye detection saves power.

[0006] In one possible implementation, the cropped N-frame images encompass the face region. The face region can include the eye region; therefore, the cropped N-frame images encompass the face region, which is beneficial for subsequent eye-opening / closing detection.

[0007] In one possible implementation, before acquiring N frames of images in response to a photo-taking operation, the method further includes: acquiring a first image captured by the camera in response to opening the camera; reducing the size of the first image to obtain a second image, the resolution of the second image being smaller than that of the first image; performing face recognition on the second image to obtain the location information of the face region in the second image; storing the first image in a first queue and storing the location information of the face region in the second image in a first buffer; acquiring N frames of images in response to a photo-taking operation includes: acquiring N frames of images from the first queue in response to a photo-taking operation; the method further includes: acquiring the location information of the face region in a target image from the first buffer in response to a photo-taking operation, the timestamp of the target image being the same as the timestamp of the N frames of images; converting the location information of the face region in the target image to obtain the location information of the face region in the N frames of images; cropping the N frames of images to obtain cropped N frames of images, including: cropping the N frames of images based on the location information of the face region in the N frames of images to obtain cropped N frames of images. In this way, since the resolution of the target image is less than that of N frames, performing face recognition on the target image helps reduce power consumption.

[0008] In one possible implementation, the first image is the original image, and the second image is the preview image, or the resolution of the second image is smaller than that of the preview image.

[0009] If the resolution of the second image is lower than that of the preview image, then performing face recognition on the second image can help reduce power consumption.

[0010] In one possible implementation, a second algorithm is used to identify the cropped N frames of images to obtain the open / closed eye results of the N frames. This includes: denoising the cropped N frames to obtain denoised images; and using the second algorithm to identify the denoised images to obtain the open / closed eye results of the N frames. Thus, performing open / closed eye detection on the denoised images helps improve the accuracy of open / closed eye detection.

[0011] In one possible implementation, the weight of the first ranking result is the first weight, and the weight of the second ranking result is the second weight, with the second weight being greater than the first weight. The fact that the second weight is greater than the first weight indicates that the eye-opening / closing detection result is more important, which helps to reduce the probability of closed eyes appearing in the fused image.

[0012] In one possible implementation, the weight of the second ranking result is positively correlated with the number of facial regions included in the N frames of images.

[0013] The weights of the second ranking result are variable; the more facial regions included in the N frames, the greater the weight of the second ranking result can be. This allows the weights of the second ranking result to change according to the number of facial regions in the N frames, providing greater flexibility and reducing the probability of closed eyes in the fused image.

[0014] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip or chip system within an electronic device. When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.), or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0015] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0017] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0018] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0019] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0020] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0021] Figure 1 A schematic diagram of a portrait photography scene provided in an embodiment of this application;

[0022] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0024] Figure 4 A schematic block diagram illustrating an image processing method provided in an embodiment of this application;

[0025] Figure 5 A schematic flowchart illustrating an image processing method provided in an embodiment of this application;

[0026] Figure 6 This is a schematic block diagram of a chip provided in an embodiment of this application. Detailed Implementation

[0027] To facilitate a clear description of the technical solutions in the embodiments of this application, the following explanation is provided first:

[0028] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, "first algorithm" and "second algorithm" are only used to distinguish different algorithms and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that terms such as "first" and "second" do not necessarily imply that they are different.

[0029] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0030] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0031] Currently, in portrait photography scenarios, when there are many people to be photographed, the images captured by electronic devices tend to show a large number of people with closed eyes, which cannot meet user needs.

[0032] For example, Figure 1 This diagram illustrates a portrait photography scenario. Figure 1 As shown, in response to the user opening the camera app, the electronic device can display... Figure 1 The interface is shown in the image. (Example: ...) Figure 1 As shown in interface a, the interface includes an image captured by the camera, a photo capture control 101, and a control 102 for viewing the captured image. The image captured by the camera includes user A and user B, both of whom are in an open-eyed state.

[0033] The electronic device responds to the user's triggering of the camera control 101, captures an image through the camera, and can display it. Figure 1 The b interface in the example. Figure 1As shown in interface b, the interface includes a thumbnail 103 of the image captured by the camera. The user can trigger a control 102 to view the captured image. In response to the user triggering the control 102 to view the captured image, the electronic device can display... Figure 1 The C interface in [the context]. For example... Figure 1 As shown in interface c, the interface includes an image captured by a camera. In the image, user A has their eyes closed, while user B has their eyes open, which does not meet user A's needs.

[0034] In some implementations, in response to a photo-taking operation, an electronic device can acquire multiple frames of raw images. The electronic device can then select a reference frame image and a suboptimal frame image from these multiple raw images based on jitter data. The reference frame image is used as the primary frame, and the suboptimal frame image is used as a supplementary frame. The reference frame image and the suboptimal frame image are then fused to obtain the final image, thereby ensuring the clarity of the photograph.

[0035] In portrait photography, when there are many people to be photographed, the probability of the acquired multi-frame original images including images of people with their eyes closed is relatively high because people blink. If the reference frame image is an image of a person with their eyes closed, the final image will likely also be an image of a person with their eyes closed, resulting in the final image not meeting the user's needs.

[0036] For example, in the above Figure 1 In the example shown, in response to the user's triggering of the camera control 101, the electronic device can capture multiple frames of raw images through the camera and select a reference frame image from these multiple frames of raw images based on jitter data. If user A is in a closed-eye state in this reference frame image, then user A is likely to be in a closed-eye state in the final image, which will not meet the user's needs.

[0037] In view of this, embodiments of this application provide an image processing method and related apparatus. In response to a photo-taking operation, multiple frames of original images can be acquired. An electronic device can select a reference frame image and a suboptimal frame image from these multiple frames of original images based on jitter data and eye-opening / closing data. The final image obtained through this reference frame image and suboptimal frame image has a relatively low probability of including a person with closed eyes, which is beneficial to reducing the probability of closed eyes in the image.

[0038] The electronic devices in this application embodiment may include handheld devices with cameras and vehicle-mounted devices including cameras, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0039] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0040] To facilitate understanding of the embodiments of this application, the hardware structure of the electronic device provided in the embodiments of this application will be described below.

[0041] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. For example... Figure 2As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, and a display screen 194, etc.

[0042] Optionally, the aforementioned sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0043] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0044] Camera 193 may include one or more cameras for capturing images. Images captured by camera 193 may be stored in internal memory 121. In this embodiment, camera 193 may be used to capture images in a portrait photography scene.

[0045] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. A layered architecture can use the Android system, the Apple iOS system, or other operating systems; this application embodiment does not limit this. The following uses a layered Android system as an example to exemplify the software architecture of the electronic device provided in this application embodiment.

[0046] Figure 3 This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application. Figure 3As shown, a layered architecture can divide the software system of an electronic device into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom: applications, application framework, hardware abstraction layer (HAL), kernel, and hardware layer.

[0047] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. For example... Figure 3 As shown, the application package can include applications such as camera and gallery.

[0048] The application framework layer provides APIs and a programming framework for applications within the application layer. The application framework layer includes predefined functions. For example... Figure 3 As shown, the application framework layer may include a camera access interface and a view system. The camera access interface can be used to provide an application programming interface and programming framework for camera applications. The view system includes visual controls, such as controls for taking pictures (e.g., those mentioned above). Figure 1 The camera control 101 shown in interface a) and the controls for viewing the captured image (such as those mentioned above) Figure 1 The control 102 shown in interface a is used to view the captured images.

[0049] like Figure 3 As shown, the HAL layer may include a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer can provide virtual hardware for the camera device. The camera algorithm library may include runtime code and data that implement the image processing methods provided in the embodiments of this application.

[0050] The kernel layer is the layer between hardware and software. For example... Figure 3 As shown, this kernel layer may include one or more of the following: camera device driver, digital signal processor driver, and image processor driver. The camera device driver is used to drive the camera's sensor to acquire images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0051] The hardware layer can include hardware such as cameras and displays.

[0052] It should be understood that in some embodiments, layers that perform the same function may be called by other names, or layers that can perform the functions of multiple layers may be considered as one layer, or layers that can perform the functions of multiple layers may be divided into multiple layers. This application does not impose any limitations on this.

[0053] The following is in conjunction with the above. Figure 3 The software structure shown below provides a detailed description of the image processing method in the embodiments of this application:

[0054] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface in the application framework layer to launch the camera application, and then sends a command to start the camera by calling the camera hardware abstraction layer. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer. The camera device driver can then start the corresponding camera to capture images.

[0055] The image processor driver can drive the graphics signal processor to reduce the preview image captured in real time by the camera to a tiny image. This tiny image can be transmitted to the camera algorithm library in real time via the image processor driver. The raw images captured in real time by the camera can be stored in a zero shutter lag (ZSL) queue. The camera algorithm library executes the image processing method provided in this application embodiment, performs face detection on the tiny image, obtains the position information of the face bounding box, and stores the position information of the face bounding box in a buffer. In response to the shooting operation, multiple frames of raw images are obtained from the ZSL queue. Reference frame images and suboptimal frame images are selected from the multiple frames of raw images based on jitter data and eye-opening / closing data. The reference frame images and suboptimal frame images are fused to obtain the final image. The camera algorithm library can transmit the final image to the camera hardware abstraction layer. The camera hardware abstraction layer can transmit it to the display screen for display. The position information of the face bounding box is used to obtain the face region from the multiple frames of images, and to perform eye-opening / closing detection on the face region to obtain eye-opening / closing data. The resolution of the tiny image is smaller than that of the original image. Therefore, performing face detection on the tiny image is more energy-efficient than performing face detection directly on the original image.

[0056] To better understand the image processing method provided in the embodiments of this application, the following will be combined with... Figure 4 Please provide a detailed explanation.

[0057] Figure 4 A schematic block diagram of an image processing method provided in an embodiment of this application is shown. Figure 4 As shown, the method includes the following steps:

[0058] S401. With the camera on, capture real-time online tiny images.

[0059] With the camera on, the electronic device can acquire the raw image captured by the camera in real time, process the raw image to obtain a preview image, and then scale down the preview image to obtain a tiny image online in real time. The resolution of the preview image is lower than that of the raw image, and the resolution of the tiny image is lower than that of the preview image.

[0060] In some examples, the original image resolution can be 4096*3072, the preview image resolution can be 1920*1080, and the tiny image resolution can be 640*480.

[0061] Alternatively, the electronic device can utilize hardware such as those described above. Figure 2 The graphics signal processor shown reduces the size of the preview image to obtain a tiny image in real-time online. This improves processing efficiency compared to software processing.

[0062] like Figure 4 As shown, the online real-time tiny map can include small... Figure 1 ,Small Figure 2 ,Small Figure 3 and small Figure 4 Images, etc.

[0063] Understandably, each tiny image in the online real-time tiny image corresponds to a timestamp, which is used to indicate the time when the tiny image was obtained.

[0064] S402. Input the real-time online tiny image into the face detection module to obtain the position information of the face bounding box.

[0065] The online real-time tiny image may include one or more faces, and this application embodiment does not limit this. If the online real-time tiny image includes one face, the face detection module can output the position information of one face bounding box; if the online real-time tiny image includes multiple faces, the face detection module can output the position information of multiple face bounding boxes.

[0066] like Figure 4 As shown, in some examples, the face detection module can output the position information of 5 face frames, including the position information of face frame 1, face frame 2, face frame 3, face frame 4, and face frame 5.

[0067] The position information of the face bounding box output by the electronic device can be stored in a buffer.

[0068] In this embodiment, since the resolution of the tiny image is smaller than that of the preview image or the original image, performing face detection on the tiny image helps to save power consumption.

[0069] S403. With the camera on, acquire the raw image captured by the camera and store the raw image in the ZSL queue.

[0070] ZSL queues can be used to store a fixed number of images that can be updated over time. For example... Figure 4 As shown, in some examples, the ZSL queue includes raw image 1 (Raw1), raw image 2 (Raw2), raw image 3 (Raw3), and raw image 4 (Raw4), etc.

[0071] S403 and S401 execute in parallel. In some implementations, S401 and S403 are implemented using different threads.

[0072] It is understandable that S401 to S403 mentioned above can be executed as soon as the camera application starts.

[0073] It is understandable that each frame of the raw image captured by the camera corresponds to a timestamp, which is used to indicate the time when the raw image frame was obtained.

[0074] S404, In response to the photo-taking operation, select multiple frames of original images from the ZSL queue through the frame selection module.

[0075] The number of original multi-frame images can be less than or equal to the number stored in the ZSL queue. The number of original multi-frame images can be preset.

[0076] like Figure 4 As shown, the number of original images in the multiple frames can be 3. RawA, RawB, and RawC are selected from Raw1, Raw2, Raw3, and Raw4. RawA can be Raw1, RawB can be Raw2, and RawC can be Raw3. Alternatively, RawA can be Raw1, RawB can be Raw2, and RawC can be Raw4. These are not all examples listed here.

[0077] This application provides several methods for selecting multiple frames of original images from the ZSL queue.

[0078] In one possible implementation, the electronic device takes the moment of taking the photo as a starting point and obtains multiple frames of raw images after that moment. This is suitable for shooting still portraits.

[0079] In another possible implementation, the electronic device uses the moment of taking the photo as the midpoint to obtain a portion of the image before the photo's capture and another portion after the photo's capture, resulting in multiple original frames. This is suitable for capturing moving portrait scenes.

[0080] S405. Based on the timestamp of each original image in the multi-frame original image, a tiny image corresponding to the timestamp can be obtained from the online real-time tiny image, realizing the matching of the tiny image with the original image, and then obtaining the position information of the face box corresponding to the tiny image.

[0081] S401 and S403 can be implemented in different threads, which may result in latency. Matching the tiny image with the original image using timestamps helps to obtain more accurate face bounding box position information.

[0082] S406. The position information of the face bounding box is transformed to obtain the transformed position information of the face bounding box, which is used to represent the position information of the face in the original image.

[0083] The original image and the tiny image have different resolutions. To obtain the position information of the face bounding box in the original image, the position information of the face bounding box in the tiny image needs to be converted.

[0084] S407. Input the multi-frame original images, the jitter amount of each frame in the multi-frame original images, and the position information of the converted face box corresponding to each frame in the multi-frame original images into the frame selection algorithm module to obtain the output of the frame selection algorithm module. The frame selection algorithm module is used to sort the multi-frame original images and output the optimal frame sequence.

[0085] like Figure 4 As shown, the frame selection algorithm module may include a jitter-based frame selection submodule and a portrait scene eye-open / closed frame selection submodule. The jitter-based frame selection submodule can sort multiple original images based on the jitter of each original image to obtain a first frame sequence. In some examples, in the first frame sequence, the original images sorted earlier are preferred over those sorted later. Alternatively, the original images sorted later are preferred over those sorted earlier. Or, the original images sorted in the middle are preferred over those sorted with cards in front and those sorted later; this embodiment does not limit this.

[0086] The amount of jitter can include gyroscope (GYRO) stabilization and / or optical image stabilization (OIS). For example... Figure 4 As shown, jitter can include GYRO and OIS.

[0087] In portrait scenes, the eye-opening / closing frame selection submodule can crop the face from each original image using the position information of the transformed face bounding box corresponding to each original image, obtaining a cropped face image. Then, it performs noise reduction on the cropped face image (e.g., using spatial domain filtering or transform domain filtering), obtaining a denoised image. Finally, it performs eye-opening / closing detection on the denoised image, obtaining the eye-opening / closing detection result. Since the original image has more pixels than the tiny image, using the original image for eye-opening / closing detection is more accurate than using the tiny image. Furthermore, cropping the face before eye-opening / closing detection improves processing efficiency and saves power compared to using the original image directly. Additionally, performing eye-opening / closing detection on the denoised image further improves recognition accuracy. The frame selection algorithm module can sort multiple original images based on the eye-opening / closing detection results of each original image to obtain a second frame sequence.

[0088] In some examples, in the second frame sequence, the original image ordered earlier is superior to the original image ordered later, or in other words, the number of open eyes in the original image ordered earlier is greater than the number of open eyes in the original image ordered later (this can apply to multiple scenarios), or the probability of open eyes in the original image ordered earlier is greater than the probability of open eyes in the original image ordered later (this can apply to a scenario with only one person).

[0089] In other examples, in the second frame sequence, the later original images are superior to the earlier original images, or in other words, the number of open eyes in the later original images is greater than the number of open eyes in the earlier original images (this can apply to multiple scenarios), or the probability of open eyes in the later original images is greater than the probability of open eyes in the earlier original images (this can apply to a scenario with only one person).

[0090] The frame selection algorithm can comprehensively evaluate the first frame sequence and the second frame sequence to output the optimal frame sequence.

[0091] For example, each original image in the first frame sequence can correspond to a first score, and each original image in the second frame sequence can correspond to a second score. The first frame sequence can correspond to a first weight, and the second frame sequence can correspond to a second weight. Based on the first score, second score, first weight, and second weight, the score of each original image can be calculated. Based on this score, the frames are sorted to obtain the optimal frame sequence. The original images sorted earlier in the optimal frame sequence are superior to the original images sorted later.

[0092] It is understood that this application embodiment focuses on capturing images without closed eyes; therefore, in this application embodiment, the weight of the second frame sequence can be greater than the weight of the first frame sequence. In other scenarios, such as night scenes, the probability of human figures appearing is relatively low, and the weight of the second frame sequence can be less than the weight of the first frame sequence. Alternatively, the weights of the first and second frame sequences can vary depending on the scene.

[0093] S408. Using the first frame of the optimal frame sequence as the reference frame and the other frames in the optimal frame sequence as the suboptimal frames, the reference frame image and the suboptimal frame image are fused to obtain the final image.

[0094] The fusion of reference frame and suboptimal frame images involves frame fusion technology, which is not limited in the embodiments of this application.

[0095] The image processing method provided in this application reduces the probability of closed eyes in the captured image by combining the jitter amount and the results of eye opening and closing detection in multiple original images to determine the reference frame and the suboptimal frame, while maintaining image stabilization.

[0096] The above combination Figure 4 The method provided by the embodiments of this application has been introduced. The method provided by the embodiments of this application will be described below in conjunction with the step flow.

[0097] For example, Figure 5 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 5 As shown, the method may include the following steps:

[0098] S501. In response to the photo-taking operation, acquire N frames of images, where N is an integer greater than 1.

[0099] The N-frame image can be referenced from the above multi-frame image. This N-frame image can be used to represent the raw image captured by the camera.

[0100] S502. Obtain the image sequence after sorting the N frames; wherein the N frames in the image sequence are sorted according to the fusion weight of the N frames when performing image fusion, and the sorting of the N frames in the image sequence is related to the first sorting result and the second sorting result. The first sorting result is obtained by sorting the N frames using the first algorithm for jitter processing, and the second sorting result is obtained by sorting the N frames using the second algorithm for eye opening and closing detection.

[0101] The fusion weight can be a value calculated based on the first and second ranking results. The first ranking result represents the ranking of N frames of images based on jitter data, and the second ranking result represents the ranking of N frames of images based on open / closed eye data. The fusion weight is related to both the jitter data and the open / closed eye data.

[0102] S503. Image fusion based on image sequences.

[0103] The image with the highest fusion weight in the image sequence can be the reference frame image, and the images in the image sequence other than the reference frame can be the suboptimal frame images. The reference frame image and the suboptimal frame images are fused to obtain the final image.

[0104] The image processing method provided in this application sorts N frames of images based on shake data and open / closed eye data, which helps to reduce the probability of closed eyes in the captured images while stabilizing the image.

[0105] Optionally, the second sorting result is obtained by cropping N frames of images to obtain cropped N frames of images, the cropped N frames of images including the eye region; using a second algorithm to identify the cropped N frames of images to obtain the open / closed eye results of the N frames of images, the open / closed eye results being used to represent the number of open eyes and / or closed eyes in the image, or to represent the probability of open eyes in the image; and sorting the N frames of images according to the open / closed eye results to obtain the second sorting result.

[0106] The cropped N-frame images include the eye region. These cropped N-frame images can be circular, elliptical, square, or rectangular regions that include the eye region, etc., and this application embodiment does not limit this. The resolution of the cropped N-frame images must be lower than the resolution of the original N-frame images, and they only need to include the eye region.

[0107] In some examples, the resolution of the eye region can be 30*30, and the resolution of the cropped N frames can be greater than 30*30.

[0108] If an image contains a large number of eyes, the open / closed eye result can be used to represent the number of open eyes and / or closed eyes in the image. For example, if an image contains 5 people, the open / closed eye result could include 3 open eyes and 2 closed eyes.

[0109] If an image contains few eyes, the open / closed eye result can be used to represent the probability that the eyes are open in the image. For example, if the image contains one person, the open / closed eye result could include a probability of 80% for the eyes being open.

[0110] In the second sorting result, the image ranked first may be superior to the image ranked last, or the image ranked last may be superior to the image ranked first, or the image ranked in the middle may be superior to the image ranked first and also superior to the image ranked last. This application does not limit this.

[0111] Since the resolution of the cropped N-frame images is lower than that of the original N-frame images, cropping the N-frame images before performing eye-opening / closing detection helps save power.

[0112] Optionally, the area included in the cropped N-frame images is the face region. The face region can include the eye region, so the area included in the cropped N-frame images is the face region, which is beneficial for subsequent eye-opening and closing detection.

[0113] Cropping N frames of images to obtain the area encompassed by the cropped N frames of images as the face region can be implemented in various ways.

[0114] In one possible implementation, face recognition is performed on N frames of images to obtain the location information of the face regions. Based on the location information of the faces, the N frames of images are then cropped to obtain N cropped frames. This approach is simple to implement.

[0115] In another possible implementation, face recognition is performed on other images related to the N-frame image to obtain the location information of the face region, where the resolution of the other images is lower than that of the N-frame image. The location information of the face region is then transformed to obtain the location information of the face region in the N-frame image. Based on the location information of the face in the N-frame image, the N-frame image is cropped to obtain the cropped N-frame image.

[0116] In this way, since the resolution of other images is lower than that of N frames, face recognition based on the resolution of other images helps to reduce power consumption.

[0117] Optionally, before acquiring N frames of images in response to a photo-taking operation, the method further includes: acquiring a first image captured by the camera in response to opening the camera; reducing the size of the first image to obtain a second image, the resolution of the second image being smaller than the resolution of the first image; performing face recognition on the second image to obtain the position information of the face region in the second image; storing the first image in a first queue and storing the position information of the face region in the second image in a first buffer; acquiring N frames of images in response to a photo-taking operation includes: acquiring N frames of images from the first queue in response to a photo-taking operation; the method further includes: acquiring the position information of the face region in a target image from the first buffer in response to a photo-taking operation, the timestamp of the target image being the same as the timestamp of the N frames of images; converting the position information of the face region in the target image to obtain the position information of the face region in the N frames of images; cropping the N frames of images to obtain cropped N frames of images, including: cropping the N frames of images based on the position information of the face region in the N frames of images to obtain cropped N frames of images.

[0118] The first image represents an image captured in real time by the camera. The electronic device can store the first image in a first queue in real time. The number of images that the first queue can store can be unlimited or finite; this embodiment does not limit this. If the number of images that can be stored in the first queue is finite, the images in the first queue can be updated over time. This helps save memory while storing the images needed. The first queue can refer to the ZSL queue described above.

[0119] The resolution of the second image is lower than that of the first image. If the first image is the original image, the second image can be a preview image or the tiny image mentioned above.

[0120] In response to the operation of opening the camera, the first image is stored in the first queue, and the first image is scaled down to obtain the second image. Face recognition is then performed on the second image, etc. These actions can be executed in parallel by different threads.

[0121] For example, the processing of the first image can refer to the above. Figure 4 The offline photo-taking process shown above, and the processing of the second image, can be referenced from the online preview process described above.

[0122] It is understandable that if the camera is on, the first and second images can be continuously generated, and the timestamps of the first and second images are the same.

[0123] The fact that the timestamp of the target image is the same as the timestamp of the N frames indicates that the target image and the N frames are identical except for the resolution. Therefore, the positional information of the face region in the target image can be converted to obtain the positional information of the face region in the N frames. This allows the N frames to be cropped based on the positional information of the face region in the N frames, resulting in the cropped N frames.

[0124] This method can refer to S401 to S406 above.

[0125] In this way, since the resolution of the target image is less than that of N frames, performing face recognition on the target image helps reduce power consumption.

[0126] Optionally, the first image is the original image, the second image is the preview image, or the resolution of the second image is lower than the resolution of the preview image.

[0127] If the resolution of the second image is smaller than the resolution of the preview image. In one example, the second image can be as described above. Figure 4 The tiny image shown. This allows for face recognition of the second image, which helps reduce power consumption.

[0128] Optionally, the second algorithm is used to identify the cropped N frames of images to obtain the open and closed eye results of the N frames of images, including: denoising the cropped N frames of images to obtain denoised images; and using the second algorithm to identify the denoised images to obtain the open and closed eye results of the N frames of images.

[0129] Therefore, performing eye-opening / closing detection on the denoised image helps improve the accuracy of eye-opening / closing detection.

[0130] Optionally, the weight of the first ranking result is the first weight, and the weight of the second ranking result is the second weight, with the second weight being greater than the first weight. A second weight greater than the first weight indicates that the eye-opening / closing detection result is more important, which helps reduce the probability of closed eyes appearing in the fused image.

[0131] For example, in portrait photography scenarios, the probability of eyes appearing in the image is relatively high. The second weight is greater than the first weight, which helps to reduce the probability of closed eyes appearing in the fused image.

[0132] Optionally, the weight of the second ranking result is positively correlated with the number of facial regions included in the N frames of images.

[0133] The weights of the second ranking result are variable; the more facial regions included in the N frames, the greater the weight of the second ranking result can be. It is understandable that each frame in the N frames includes the same number of facial regions.

[0134] For example, if each frame of N frames contains 5 facial regions, the weight of the second ranking result can be 90%; if each frame of N frames contains 3 facial regions, the weight of the second ranking result can be 80%.

[0135] In this way, the weights of the second sorting result can be changed according to the number of facial regions in the N frames of images, which is more flexible and also helps to reduce the probability of closed eyes in the fused image.

[0136] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0138] The image processing method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above image processing method.

[0139] Figure 6 This is a schematic diagram of a chip structure provided in an embodiment of this application. Figure 6 As shown, chip 60 includes one or more processors 601, communication lines 602, communication interfaces 603, and memory 604.

[0140] In some implementations, memory 604 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0141] The image processing method described in the embodiments of this application can be applied to, or implemented by, processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the image processing method can be completed by integrated logic circuits in the hardware of processor 601 or by instructions in software form. Processor 601 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 601 can implement or execute the processing-related methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0142] The steps of the image processing method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 604, and processor 601 reads the information in memory 604 and, in conjunction with its hardware, completes the steps of the above method.

[0143] The processor 601, memory 604 and communication interface 603 can communicate with each other through communication line 602.

[0144] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0145] The image processing method provided in this application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can be referred to the above-described related descriptions, which will not be repeated here.

[0146] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0147] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0148] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0149] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0150] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: In response to the operation of opening the camera, the first image captured by the camera is acquired; The first image is reduced in size to obtain a second image, the resolution of the second image being smaller than that of the first image; Perform face recognition on the second image to obtain the location information of the face region in the second image; The first image is stored in a first queue, and the location information of the face region in the second image is stored in a first buffer. In response to a photo-taking operation, N frames of images are obtained from the first queue, where N is an integer greater than 1; the position information of the face region in the target image is obtained from the first buffer, where the timestamp of the target image is the same as the timestamp of the corresponding frame image in the N frames; the position information of the face region in the target image is converted to obtain the position information of the face region in the N frames; An image sequence is obtained after sorting the N frames; wherein the N frames in the image sequence are sorted according to the fusion weights of the N frames during image fusion, and the sorting of the N frames in the image sequence is related to a first sorting result and a second sorting result. The first sorting result is obtained by sorting the N frames using a first algorithm for jitter processing. Each frame in the first sorting result has a corresponding first weight, and each frame in the second sorting result has a corresponding second weight. The second weight of each frame is greater than its corresponding first weight. The second weight of the second sorting result is positively correlated with the number of facial regions included in the corresponding frame image. Image fusion is performed based on the image sequence; The second sorting result was obtained in the following way: The N frames of images are cropped based on the position information of the face region in the N frames of images to obtain the cropped N frames of images, and the cropped N frames of images include the eye region; The second algorithm is used to identify the cropped N frames of images to obtain the open and closed eye results of the N frames of images. The open and closed eye results are used to represent the number of open eyes and / or closed eyes in the image, or to represent the probability of open eyes in the image. Based on the eye-opening and eye-closing results of the N frames of images, the N frames of images are sorted to obtain the second sorting result.

2. The method according to claim 1, characterized in that, The area included in the cropped N-frame image is the face region.

3. The method according to any one of claims 1 to 2, characterized in that, The step of using the second algorithm to identify the cropped N frames of images and obtain the open / closed eye results of the N frames includes: The cropped N frames of images are denoised to obtain the denoised image; The second algorithm is used to identify the denoised image to obtain the open / closed eye results of the N frames.

4. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1-3.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

6. A chip system, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being used to run a computer program or instructions to perform the method as described in any one of claims 1-3.

7. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1-3.

Images