Image processing method and apparatus, storage medium, and electronic device

Abstract

The present application relates to the technical field of image processing, and discloses an image processing method and apparatus, a storage medium, and an electronic device. The method comprises: selecting a basic image of which image quality information meets a preset quality condition; selecting an eye image of which eye-opening state information is matched with that of the basic image; and on the basis of the basic image and the eye image, performing multi-frame fusion processing on a plurality of images to be processed to obtain a processed image. The present application can improve the eye-opening effect of images, and improve the user experience.

Description

Image processing method, device, storage medium and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 5, 2024, with application number 2024100232591 and application name “Image processing method, device, storage medium and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0003] In scenes such as taking photos, with the continuous improvement of image processing technology, users have higher and higher requirements for images. Many shooting devices also provide image processing methods to improve image effects. However, due to various factors, a large number of subjects will naturally blink when taking photos, which often results in photos with closed eyes or other eye-opening effects that are not good. Technical issues

[0004] Current image processing methods are often unable to effectively process such images, and such images are often treated as waste, affecting user experience. Technical Solutions

[0005] The embodiment of the present application provides an image processing solution that can effectively improve the eye-opening effect of the image and enhance the user experience.

[0006] The embodiments of this application provide the following technical solutions:

[0007] According to one embodiment of the present application, an image processing method includes: performing image detection on the multiple images to be processed to obtain image quality information and eye-opening state information corresponding to each of the images to be processed; selecting an image to be processed whose image quality information meets predetermined quality conditions from the multiple images to be processed to obtain a basic image; selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image; and performing multi-frame fusion processing on the multiple images to be processed based on the basic image and the eye image to obtain a processed image.

[0008] In some embodiments of the present application, based on the basic image and the eye image, the multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image, including: if the basic image and the eye image are not the same image, the multiple images to be processed are subjected to multi-frame noise reduction synthesis using the basic image as the basic frame to obtain a first fused noise-reduced image; and the eye area in the eye image is fused into the first fused noise-reduced image to obtain the processed image.

[0009] In some embodiments of the present application, the eye-opening state information is information used to describe whether the eyes are in an open state; the step of selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image includes: selecting an image to be processed in which the eyes are in an open state from the multiple images to be processed according to the eye-opening state information corresponding to each image to be processed; and selecting the image with the highest image quality from the images to be processed in which the eyes are in an open state to obtain the eye image.

[0010] In some embodiments of the present application, the eye-opening state information is information used to describe the degree of eye openness and the eye-opening posture; the step of selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image includes: performing object feature extraction processing on the basic image to obtain object feature data; obtaining object state information based on the object feature data, the object state information being information used to describe the object posture of the target object; matching and analyzing the eye-opening state information of each image to be processed with the object state information to obtain the eye image, the eye image being the image to be processed corresponding to the eye-opening state information that matches the object state information.

[0011] In some embodiments of the present application, the matching analysis of the eye-opening state information of each image to be processed with the object state information to obtain the eye image includes: performing state combination scoring processing on the eye-opening state information of each image to be processed and the object state information respectively to obtain a state fusion score corresponding to each image to be processed, wherein the state fusion score is a state fusion score of the eye in the image to be processed and the target object in the basic image; and selecting an image to be processed according to the state fusion score corresponding to each image to be processed to obtain the eye image.

[0012] In some embodiments of the present application, the object state information includes facial state information and body state information of the target object; the state combination scoring processing of the eye-opening state information of each image to be processed and the object state information to obtain a state fusion score corresponding to each image to be processed includes: performing state combination scoring of the eye-opening state information of each image to be processed and the facial state information to obtain a first score corresponding to each image to be processed; performing state combination scoring of the eye-opening state information of each image to be processed and the body state information to obtain a second score corresponding to each image to be processed; and obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed.

[0013] In some embodiments of the present application, obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed includes: obtaining a weighting coefficient for matching the object state information based on the object state information; and weightedly summing the first score and the second score corresponding to each image to be processed according to the weighting coefficient to obtain a state fusion score corresponding to each image to be processed.

[0014] In some embodiments of the present application, based on the basic image and the eye image, the multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image, including: if the basic image and the eye image are the same image, the multiple images to be processed are subjected to multi-frame noise reduction synthesis with the basic image as the basic frame to obtain a second fused denoised image; and the processed image is obtained based on the second fused denoised image.

[0015] In some embodiments of the present application, the selecting of the images to be processed whose image quality information meets the predetermined quality conditions from the multiple images to be processed to obtain the basic image includes one of the following methods: selecting the image to be processed with the highest image quality from the multiple images to be processed based on the image quality information in each of the images to be processed to obtain the basic image; selecting one or more images to be processed whose image quality is higher than a predetermined quality threshold from the multiple images to be processed based on the image quality information in each of the images to be processed to obtain the basic image.

[0016] In some embodiments of the present application, the image quality information includes but is not limited to one or more of image clarity, resolution, contrast, brightness, noise level, color balance, image distortion level, and distortion level.

[0017] According to one embodiment of the present application, an image processing device includes: a detection module for performing image detection on the multiple images to be processed to obtain image quality information and eye-opening state information corresponding to each of the images to be processed; a selection module for selecting an image to be processed whose image quality information meets a predetermined quality condition from the multiple images to be processed to obtain a basic image; a matching module for selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image; and a fusion module for performing multi-frame fusion processing on the multiple images to be processed based on the basic image and the eye image to obtain a processed image.

[0018] In some embodiments of the present application, the fusion module is used to: if the base image and the eye image are not the same image, perform multi-frame noise reduction synthesis on the multiple images to be processed using the base image as the base frame to obtain a first fused noise-reduced image; and fuse the eye area in the eye image into the first fused noise-reduced image to obtain the processed image.

[0019] In some embodiments of the present application, the eye-opening state information is information used to describe whether the eyes are in an open state; the matching module is used to: select an image to be processed in which the eyes are in an open state from the multiple images to be processed according to the eye-opening state information corresponding to each image to be processed; select the image with the highest image quality from the images to be processed in which the eyes are in an open state to obtain the eye image.

[0020] In some embodiments of the present application, the eye-opening state information is information used to describe the degree of eye openness and the eye-opening posture; the matching module is used to: perform object feature extraction processing on the basic image to obtain object feature data; obtain object state information based on the object feature data, and the object state information is information used to describe the object posture of the target object; match and analyze the eye-opening state information of each image to be processed with the object state information to obtain the eye image, and the eye image is the image to be processed corresponding to the eye-opening state information that matches the object state information.

[0021] In some embodiments of the present application, the matching module is used to: perform state combination scoring processing on the eye-opening state information of each image to be processed and the object state information, respectively, to obtain a state fusion score corresponding to each image to be processed, wherein the state fusion score is the state fusion score of the eye in the image to be processed and the target object in the basic image; select an image to be processed according to the state fusion score corresponding to each image to be processed to obtain the eye image.

[0022] In some embodiments of the present application, the object state information includes facial state information and body state information of the target object; the matching module is used to: perform a state combination score on the eye-opening state information of each image to be processed and the facial state information, to obtain a first score corresponding to each image to be processed; perform a state combination score on the eye-opening state information of each image to be processed and the body state information, to obtain a second score corresponding to each image to be processed; and obtain a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed.

[0023] In some embodiments of the present application, the matching module is used to: obtain a weighted coefficient for matching the object state information based on the object state information; and perform weighted summation of the first score and the second score corresponding to each image to be processed according to the weighted coefficient to obtain a state fusion score corresponding to each image to be processed.

[0024] In some embodiments of the present application, the fusion module is used to: if the basic image and the eye image are the same image, perform multi-frame noise reduction synthesis on the multiple images to be processed using the basic image as the basic frame to obtain a second fused noise-reduced image; and obtain the processed image based on the second fused noise-reduced image.

[0025] In some embodiments of the present application, the selection module is used to implement one of the following methods: based on the image quality information in each of the images to be processed, selecting an image to be processed with the highest image quality from the multiple images to be processed to obtain the basic image; based on the image quality information in each of the images to be processed, selecting one or more images to be processed with image quality higher than a predetermined quality threshold from the multiple images to be processed to obtain the basic image.

[0026] In some embodiments of the present application, the image quality information includes but is not limited to one or more of image clarity, resolution, contrast, brightness, noise level, color balance, image distortion level, and distortion level.

[0027] According to another embodiment of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method described in the embodiment of the present application.

[0028] According to another embodiment of the present application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the method described in the embodiment of the present application.

[0029] According to another embodiment of the present application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations described in the embodiments of the present application. Beneficial effects

[0030] In an embodiment of the present application, image detection is performed on the multiple images to be processed to obtain image quality information and eye-opening status information corresponding to each of the images to be processed; an image to be processed whose image quality information meets predetermined quality conditions is selected from the multiple images to be processed to obtain a basic image; an image to be processed whose eye-opening status information matches the basic image is selected from the multiple images to be processed to obtain an eye image; based on the basic image and the eye image, the multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image.

[0031] In this way, first, an image to be processed whose image quality information meets predetermined quality conditions is selected from multiple images to be processed as a basic image, and then, an image to be processed whose eye-opening state information matches the basic image is selected as the eye image. Based on the basic image and the eye image, multiple frames of the multiple images to be processed are fused to obtain a processed image. Since the eye-opening state information of the eye image matches the basic image, the eye-opening effect of the processed image obtained by multi-frame fusion will be very good, which can effectively improve the eye-opening effect of the image and enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0033] FIG1 shows a flowchart of an image processing method according to an embodiment of the present application.

[0034] FIG2 shows a flowchart of image selection according to an embodiment of the present application.

[0035] FIG3 shows an image selection flowchart according to another embodiment of the present application.

[0036] FIG4 shows a block diagram of an image processing apparatus according to an embodiment of the present application.

[0037] FIG5 shows a block diagram of an electronic device according to an embodiment of the present application.

[0038] Implementation Methods of the Application

[0039] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the examples provided herein are merely for explaining the present disclosure and are not intended to limit the present disclosure. In addition, the examples provided below are partial examples for implementing the present disclosure, rather than providing all examples for implementing the present disclosure. In the absence of conflict, the technical solutions described in the examples of the present disclosure may be implemented in any combination.

[0040] It should be noted that, in the embodiments of the present disclosure, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus comprising a series of elements includes not only the elements explicitly stated, but also other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other related elements (such as steps in the method or units in the apparatus, for example, a unit may be part of a circuit, part of a processor, part of a program or software, etc.) in the method or apparatus comprising the element.

[0041] For example, the image processing method provided by the embodiment of the present disclosure includes a series of steps, but the image processing method provided by the embodiment of the present disclosure is not limited to the recorded steps. Similarly, the image processing device provided by the embodiment of the present disclosure includes a series of units, but the device provided by the embodiment of the present disclosure is not limited to including the units explicitly recorded, and may also include units that need to be set up to obtain relevant information or perform processing based on information.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure.

[0043] Figure 1 schematically illustrates a flow chart of an image processing method according to an embodiment of the present application. The image processing method can be executed by any device or server with processing capabilities, such as a television, computer, mobile phone, smartwatch, or home appliance, and a server such as a cloud server or a physical server.

[0044] As shown in FIG. 1 , the image processing method may include steps S110 to S140 .

[0045] Step S110, performing image detection on the multiple images to be processed to obtain image quality information and eye-opening status information corresponding to each of the images to be processed; Step S120, selecting an image to be processed whose image quality information meets a predetermined quality condition from the multiple images to be processed to obtain a basic image; Step S130, selecting an image to be processed whose eye-opening status information matches the basic image from the multiple images to be processed to obtain an eye image; Step S140, performing multi-frame fusion processing on the multiple images to be processed based on the basic image and the eye image to obtain a processed image.

[0046] The multiple images to be processed can be multiple images (e.g., 6 images) in an image sequence obtained by real-time photography. The multiple images to be processed can also be multiple consecutive images stored in a predetermined location rather than obtained by real-time photography. The number of images to be processed can be set according to actual circumstances.

[0047] After acquiring multiple images to be processed, image detection can be performed on each of the images to be processed to obtain image quality information and eye-opening status information corresponding to each image to be processed. For example, image detection can be performed on each of the images to be processed using a preset image detection neural network.

[0048] Image quality information is information that can reflect image quality and may include, but is not limited to, one or more of image clarity, resolution, contrast, brightness, noise level, color balance, image distortion, and degree of distortion. In one embodiment of the present application, the image quality information specifically refers to image clarity. Eye openness information is information that can reflect the openness of eyes in an image and may include information such as whether the eyes are open, the degree of eye openness, or an eye openness score.

[0049] From a plurality of images to be processed, an image to be processed whose image quality information meets a predetermined quality condition is selected to obtain a basic image. For example, an image to be processed with the highest definition can be selected as the basic image.

[0050] Furthermore, from multiple images to be processed, an image to be processed whose eye-opening state information matches the basic image can be selected to obtain an eye image. It can be understood that the basic image and the eye image may be the same image or may not be the same image.

[0051] Based on the basic image and the eye image, multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image. This can usually ensure that the image quality of the processed image is high and the eyes in the image are open. On this basis, the state of the open eyes in the processed image will be more coordinated than in related technologies.

[0052] In this way, based on steps S110 to S140, first, an image to be processed with the highest image quality is selected from multiple images to be processed as a basic image, and then, an image to be processed whose eye-opening state information matches the basic image is selected as an eye image. Based on the basic image and the eye image, multiple frames of multiple images to be processed are fused to obtain a processed image. Since the eye-opening state information of the eye image matches the basic image, the eye-opening effect of the processed image obtained by multi-frame fusion will be very good, which can effectively improve the eye-opening effect of the image and enhance the user experience.

[0053] The following describes further optional specific embodiments of each step performed when performing image processing in the embodiment of FIG. 1 .

[0054] In one embodiment, selecting an image to be processed whose image quality information meets a predetermined quality condition from among the plurality of images to be processed to obtain a base image includes one of the following methods:

[0055] The first method is to select an image to be processed with the highest image quality from the multiple images to be processed according to the image quality information of each image to be processed, to obtain the basic image;

[0056] The second method is to select one or more images to be processed whose image quality is higher than a predetermined quality threshold from the multiple images to be processed according to the image quality information of each image to be processed, to obtain the basic image.

[0057] In the first approach, the highest-quality image is selected from among multiple images to be processed as the base image. Specifically, this highest-quality image can be the one with the highest image quality, as reflected by clarity or other image quality information. Selecting the highest-quality image effectively ensures the quality of the processed image obtained through fusion.

[0058] In the second method, one or more images to be processed whose image quality exceeds a predetermined quality threshold are selected from multiple images to be processed as the base image. Thus, the base image may be one or more, and the predetermined quality threshold can be set based on actual conditions. If there is only one base image, a single processed image is ultimately generated. If there are multiple base images, steps S130 and S140 are performed separately for each base image, resulting in multiple processed images.

[0059] In one embodiment, referring to Figure 2, the eye-opening state information is information used to describe whether the eyes are in an open state; the step of selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image may include: step S210, selecting an image to be processed in which the eyes are in an open state from the multiple images to be processed according to the eye-opening state information corresponding to each image to be processed; step S220, selecting the image with the highest image quality from the images to be processed in which the eyes are in an open state to obtain the eye image.

[0060] In this embodiment, from multiple images to be processed, the one with the highest image quality (i.e., the image quality reflected by the image quality information) among the images to be processed in which the eyes are open is selected as the eye image, which can effectively ensure the quality, eye opening rate and coordination of the processed image obtained by fusion.

[0061] Specifically, the eye-opening state information is information describing whether the eyes are open. For example, the eye-opening state information can include information such as whether the eyes are open or closed. First, based on the eye-opening state information corresponding to each image to be processed, an image to be processed in which the eyes are open can be selected from multiple images to be processed. Furthermore, the image with the highest image quality among the images to be processed in which the eyes are open can be selected as the resulting eye image.

[0062] In one embodiment, referring to Figure 3, the eye-opening state information is information used to describe the degree of eye openness and the eye-opening posture; the step of selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image may include: step S310, performing object feature extraction processing on the basic image to obtain object feature data; step S320, obtaining object state information based on the object feature data, wherein the object state information is information used to describe the object posture of the target object; step S330, matching and analyzing the eye-opening state information of each image to be processed with the object state information to obtain the eye image, wherein the eye image is the image to be processed corresponding to the eye-opening state information that matches the object state information.

[0063] In this embodiment, from multiple images to be processed, the image to be processed corresponding to the eye-opening state information that matches the object state information in the basic image is selected as the eye image, which can effectively ensure the quality and eye-opening rate of the processed image obtained by fusion, and further effectively improve the eye-opening coordination.

[0064] The eye-opening state information is information used to describe the degree of eye opening and the eye-opening posture. For example, one type of eye-opening state information includes the eye-opening degree A1 and the eye-opening posture B1, and another type of eye-opening state information includes the eye-opening degree A2 and the eye-opening posture B3.

[0065] Using a preset feature extraction neural network, object feature extraction processing can be performed on the base image to obtain object feature data of the target object (i.e., a person, animal, or other object in the base image). Furthermore, using a preset state analysis neural network, object feature data can be analyzed to obtain object state information, which is information used to describe the object posture of the target object.

[0066] Furthermore, the eye-opening state information of each image to be processed is matched and analyzed with the object state information to obtain an eye image, which is the image to be processed corresponding to the eye-opening state information that matches the object state information.

[0067] For example, the eye-opening state information of the image to be processed T1 is matched and analyzed with the object state information T0 of the basic image to obtain the matching analysis result corresponding to the image to be processed T1; the eye-opening state information of the image to be processed T2 is matched and analyzed with the object state information T0 of the basic image to obtain the matching analysis result corresponding to the image to be processed T2, and so on, to obtain the matching analysis results corresponding to all the images to be processed. Therefore, based on all the matching analysis results, the "image to be processed corresponding to the eye-opening state information that matches the object state information of the basic image" can be determined as the eye image.

[0068] Furthermore, in one embodiment, matching and analyzing the eye-opening state information of each image to be processed with the object state information to obtain the eye image may include:

[0069] The eye-opening state information of each image to be processed and the object state information are respectively subjected to state combination scoring processing to obtain a state fusion score corresponding to each image to be processed, where the state fusion score is the state fusion score of the eyes in the image to be processed and the target object in the basic image; an image to be processed is selected according to the state fusion score corresponding to each image to be processed to obtain the eye image.

[0070] In this embodiment, during the matching analysis, a preset state fusion scoring neural network can be specifically used to perform state combination scoring processing on the eye-opening state information and the object state information of each image to be processed, and obtain a state fusion score corresponding to each image to be processed. The state fusion score is the state fusion score of the eyes in the image to be processed and the target object in the basic image. The state fusion score is used as a matching analysis result. The higher the state fusion score, the better the state fusion effect of the eyes in the corresponding image to be processed and the target object in the basic image.

[0071] For example, the eye-opening state information of the image to be processed T1 and the object state information T0 of the basic image are subjected to state combination scoring processing to obtain the state fusion score corresponding to the image to be processed T1; the eye-opening state information of the image to be processed T2 and the object state information T0 of the basic image are subjected to state combination scoring processing to obtain the state fusion score corresponding to the image to be processed T2, and so on, to obtain the state fusion scores corresponding to all the images to be processed.

[0072] An image to be processed is selected according to the state fusion score corresponding to each image to be processed as the obtained eye image. Specifically, the image to be processed with the highest state fusion score can be selected as the obtained eye image. The eye-opening state in the processed image obtained by multi-frame fusion based on the eye image is more coordinated, further improving the image fusion effect.

[0073] Furthermore, in one embodiment, the object state information includes facial state information and body state information of the target object; the state combination scoring processing of the eye-opening state information of each image to be processed and the object state information to obtain a state fusion score corresponding to each image to be processed includes: performing state combination scoring of the eye-opening state information of each image to be processed and the facial state information to obtain a first score corresponding to each image to be processed; performing state combination scoring of the eye-opening state information of each image to be processed and the body state information to obtain a second score corresponding to each image to be processed; and obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed.

[0074] In this embodiment, the object state information includes facial state information and body state information of the target object. The facial state information reflects the target object's facial expression or posture, and the body state information reflects the target object's overall body posture or posture. A state fusion score is generated based on the scores of the facial state information and the body state information, which can further improve the state fusion score's accuracy in reflecting the fusion effect between the eyes in the processed image and the target object in the base image, further enhancing the image fusion effect.

[0075] Furthermore, the eye-opening state information of each image to be processed is respectively combined with the facial state information to perform a state combination score to obtain a first score corresponding to each image to be processed; then, the eye-opening state information of each image to be processed is respectively combined with the body state information to perform a state combination score to obtain a second score corresponding to each image to be processed; then, based on the first score and the second score corresponding to each image to be processed, a state fusion score corresponding to each image to be processed is comprehensively obtained.

[0076] For example, the eye-opening state information of the image to be processed T1 and the facial state information T0-1 of the base image are combined and scored to obtain a first score T1-P1 corresponding to the image to be processed T1. The eye-opening state information of the image to be processed T1 and the body state information T0-2 of the base image are combined and scored to obtain a second score T1-P2 corresponding to the image to be processed T1. Then, based on the first score T1-P1 and the second score T1-P2, a state fusion score T1-P corresponding to the image to be processed T1 is obtained. This process is repeated in this way to obtain the state fusion scores corresponding to all the images to be processed.

[0077] In other embodiments, the object state information may only include the facial state information of the target object; the state combination scoring processing of the eye-opening state information of each image to be processed and the object state information to obtain the state fusion score corresponding to each image to be processed includes: performing state combination scoring of the eye-opening state information of each image to be processed and the facial state information to obtain a first score corresponding to each image to be processed; and obtaining a state fusion score corresponding to each image to be processed based on the first score corresponding to each image to be processed (for example, the first score corresponding to each image to be processed can be used as the state fusion score corresponding to each image to be processed).

[0078] Furthermore, obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed may specifically include: obtaining a weighting coefficient for matching the object state information based on the object state information; and performing weighted summation of the first score and the second score corresponding to each image to be processed based on the weighting coefficient to obtain a state fusion score corresponding to each image to be processed.

[0079] For different object state information, corresponding weighting coefficients can be pre-set. According to the object state information, the weighting coefficient matching the object state information is obtained. According to the weighting coefficient, the first score and the second score corresponding to each image to be processed are weighted and summed respectively to obtain the state fusion score corresponding to each image to be processed. This can further effectively ensure the accuracy of the state fusion score in reflecting the state fusion effect of the eyes in the image to be processed and the target object in the basic image.

[0080] In some other embodiments, obtaining the state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed may include: adding or multiplying the first score and the second score corresponding to each image to be processed to obtain the state fusion score corresponding to each image to be processed.

[0081] In some other embodiments, obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed may include: obtaining a unified weighting coefficient; and performing a weighted summation of the first score and the second score corresponding to each image to be processed based on the unified weighting coefficient to obtain the state fusion score corresponding to each image to be processed. The unified weighting coefficient is a weighting coefficient that is uniformly set for all situations without distinguishing between object state information.

[0082] In one embodiment, based on the basic image and the eye image, the multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image, including: if the basic image and the eye image are not the same image, the multiple images to be processed are subjected to multi-frame noise reduction synthesis using the basic image as the basic frame to obtain a first fused noise-reduced image; and the eye area in the eye image is fused into the first fused noise-reduced image to obtain the processed image.

[0083] If the base image and the eye image are different images, multiple frames of the processed images are first subjected to multi-frame noise reduction synthesis using the base image as the base frame to obtain a first fused noise-reduced image. Then, after subtracting the eye region from the eye image, the eye region in the eye image is fused to the eye region in the first fused noise-reduced image to obtain a processed image. The multi-frame noise reduction synthesis can be achieved using existing multi-frame fusion noise reduction methods.

[0084] In one embodiment, based on the basic image and the eye image, the multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image, which may include: if the basic image and the eye image are the same image, the multiple images to be processed are subjected to multi-frame noise reduction synthesis using the basic image as the basic frame to obtain a second fused noise-reduced image; and the processed image is obtained based on the second fused noise-reduced image.

[0085] If the base image and the eye image are the same image selected, multiple images to be processed are subjected to multi-frame noise reduction synthesis using the base image as the base frame to obtain a second fused noise-reduced image. Further, a processed image can be obtained based on the second fused noise-reduced image.

[0086] The multi-frame noise reduction synthesis method can be implemented using existing multi-frame fusion noise reduction methods. The processed image is obtained based on the second fused noise reduction image, and the second fused noise reduction image can be directly used as the processed image, or the second fused noise reduction image can be processed, such as by cropping, to obtain the processed image.

[0087] To facilitate better implementation of the image processing method provided in the embodiments of this application, the embodiments of this application also provide an image processing device based on the aforementioned image processing method. The meanings of the terms herein are the same as those in the aforementioned image processing method. For specific implementation details, please refer to the description in the method embodiments. Figure 4 shows a block diagram of an image processing device according to one embodiment of the present application.

[0088] As shown in Figure 4, the image processing device 400 may include: a detection module 410 can be used to perform image detection on the multiple images to be processed to obtain image quality information and eye-opening status information corresponding to each of the images to be processed; a selection module 420 can be used to select the images to be processed whose image quality information meets the predetermined quality conditions among the multiple images to be processed to obtain a basic image; a matching module 430 can be used to select an image to be processed whose eye-opening status information matches the basic image from the multiple images to be processed to obtain an eye image; a fusion module 440 can be used to perform multi-frame fusion processing on the multiple images to be processed based on the basic image and the eye image to obtain a processed image.

[0089] In some embodiments of the present application, the fusion module is used to: if the base image and the eye image are not the same image, perform multi-frame noise reduction synthesis on the multiple images to be processed using the base image as the base frame to obtain a first fused noise-reduced image; and fuse the eye area in the eye image into the first fused noise-reduced image to obtain the processed image.

[0090] In some embodiments of the present application, the eye-opening state information is information used to describe whether the eyes are in an open state; the matching module is used to: select an image to be processed in which the eyes are in an open state from the multiple images to be processed according to the eye-opening state information corresponding to each image to be processed; select the image with the highest image quality from the images to be processed in which the eyes are in an open state to obtain the eye image.

[0091] In some embodiments of the present application, the eye-opening state information is information used to describe the degree of eye openness and the eye-opening posture; the matching module is used to: perform object feature extraction processing on the basic image to obtain object feature data; obtain object state information based on the object feature data, and the object state information is information used to describe the object posture of the target object; match and analyze the eye-opening state information of each image to be processed with the object state information to obtain the eye image, and the eye image is the image to be processed corresponding to the eye-opening state information that matches the object state information.

[0092] In some embodiments of the present application, the matching module is used to: perform state combination scoring processing on the eye-opening state information of each image to be processed and the object state information, respectively, to obtain a state fusion score corresponding to each image to be processed, wherein the state fusion score is the state fusion score of the eye in the image to be processed and the target object in the basic image; select an image to be processed according to the state fusion score corresponding to each image to be processed to obtain the eye image.

[0093] In some embodiments of the present application, the object state information includes facial state information and body state information of the target object; the matching module is used to: perform a state combination score on the eye-opening state information of each image to be processed and the facial state information, to obtain a first score corresponding to each image to be processed; perform a state combination score on the eye-opening state information of each image to be processed and the body state information, to obtain a second score corresponding to each image to be processed; and obtain a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed.

[0094] In some embodiments of the present application, the matching module is used to: obtain a weighted coefficient for matching the object state information based on the object state information; and perform weighted summation of the first score and the second score corresponding to each image to be processed according to the weighted coefficient to obtain a state fusion score corresponding to each image to be processed.

[0095] In some embodiments of the present application, the fusion module is used to: if the basic image and the eye image are the same image, perform multi-frame noise reduction synthesis on the multiple images to be processed using the basic image as the basic frame to obtain a second fused noise-reduced image; and obtain the processed image based on the second fused noise-reduced image.

[0096] In some embodiments of the present application, the selection module is used to implement one of the following methods: based on the image quality information in each of the images to be processed, selecting an image to be processed with the highest image quality from the multiple images to be processed to obtain the basic image; based on the image quality information in each of the images to be processed, selecting one or more images to be processed with image quality higher than a predetermined quality threshold from the multiple images to be processed to obtain the basic image.

[0097] In some embodiments of the present application, the image quality information includes but is not limited to one or more of image clarity, resolution, contrast, brightness, noise level, color balance, image distortion level, and distortion level.

[0098] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0099] In addition, an embodiment of the present application further provides an electronic device, as shown in FIG5 . FIG5 shows a block diagram of an electronic device according to an embodiment of the present application. Specifically:

[0100] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will appreciate that the electronic device structure shown in FIG5 does not limit the electronic device and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0101] Processor 501 is the control center of the electronic device. It utilizes various interfaces and circuits to connect the various components of the entire computer device. By running or executing software programs and / or modules stored in memory 502 and accessing data stored in memory 502, it performs various computer device functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, processor 501 may include one or more processing cores; preferably, processor 501 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interfaces, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 501.

[0102] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0103] The electronic device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0104] The electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0105] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device will load the executable files corresponding to one or more computer program processes into the memory 502 according to the following instructions, and the processor 501 will run the computer program stored in the memory 502, thereby realizing the various functions of the aforementioned embodiments of the present application. For example, the processor 501 may perform the following steps:

[0106] Perform image detection on the multiple images to be processed to obtain image quality information and eye-opening status information corresponding to each of the images to be processed; select an image to be processed whose image quality information meets predetermined quality conditions from the multiple images to be processed to obtain a basic image; select an image to be processed whose eye-opening status information matches the basic image from the multiple images to be processed to obtain an eye image; based on the basic image and the eye image, perform multi-frame fusion processing on the multiple images to be processed to obtain a processed image.

[0107] In some embodiments of the present application, based on the basic image and the eye image, the multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image, including: if the basic image and the eye image are not the same image, the multiple images to be processed are subjected to multi-frame noise reduction synthesis using the basic image as the basic frame to obtain a first fused noise-reduced image; and the eye area in the eye image is fused into the first fused noise-reduced image to obtain the processed image.

[0108] In some embodiments of the present application, the eye-opening state information is information used to describe whether the eyes are in an open state; the step of selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image includes: selecting an image to be processed in which the eyes are in an open state from the multiple images to be processed according to the eye-opening state information corresponding to each image to be processed; and selecting the image with the highest image quality from the images to be processed in which the eyes are in an open state to obtain the eye image.

[0109] In some embodiments of the present application, the eye-opening state information is information used to describe the degree of eye openness and the eye-opening posture; the step of selecting an image to be processed whose eye-opening state information matches the basic image from the multiple images to be processed to obtain an eye image includes: performing object feature extraction processing on the basic image to obtain object feature data; obtaining object state information based on the object feature data, the object state information being information used to describe the object posture of the target object; matching and analyzing the eye-opening state information of each image to be processed with the object state information to obtain the eye image, the eye image being the image to be processed corresponding to the eye-opening state information that matches the object state information.

[0110] In some embodiments of the present application, the matching analysis of the eye-opening state information of each image to be processed with the object state information to obtain the eye image includes: performing state combination scoring processing on the eye-opening state information of each image to be processed and the object state information respectively to obtain a state fusion score corresponding to each image to be processed, wherein the state fusion score is a state fusion score of the eye in the image to be processed and the target object in the basic image; and selecting an image to be processed according to the state fusion score corresponding to each image to be processed to obtain the eye image.

[0111] In some embodiments of the present application, the object state information includes facial state information and body state information of the target object; the state combination scoring processing of the eye-opening state information of each image to be processed and the object state information to obtain a state fusion score corresponding to each image to be processed includes: performing state combination scoring of the eye-opening state information of each image to be processed and the facial state information to obtain a first score corresponding to each image to be processed; performing state combination scoring of the eye-opening state information of each image to be processed and the body state information to obtain a second score corresponding to each image to be processed; and obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed.

[0112] In some embodiments of the present application, obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed includes: obtaining a weighting coefficient for matching the object state information based on the object state information; and weightedly summing the first score and the second score corresponding to each image to be processed according to the weighting coefficient to obtain a state fusion score corresponding to each image to be processed.

[0113] In some embodiments of the present application, based on the basic image and the eye image, the multiple images to be processed are subjected to multi-frame fusion processing to obtain a processed image, including: if the basic image and the eye image are the same image, the multiple images to be processed are subjected to multi-frame noise reduction synthesis with the basic image as the basic frame to obtain a second fused denoised image; and the processed image is obtained based on the second fused denoised image.

[0114] In some embodiments of the present application, the selecting of the images to be processed whose image quality information meets the predetermined quality conditions from the multiple images to be processed to obtain the basic image includes one of the following methods: selecting the image to be processed with the highest image quality from the multiple images to be processed based on the image quality information in each of the images to be processed to obtain the basic image; selecting one or more images to be processed whose image quality is higher than a predetermined quality threshold from the multiple images to be processed based on the image quality information in each of the images to be processed to obtain the basic image.

[0115] In some embodiments of the present application, the image quality information includes but is not limited to one or more of image clarity, resolution, contrast, brightness, noise level, color balance, image distortion level, and distortion level.

[0116] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0117] To this end, an embodiment of the present application further provides a storage medium storing a computer program, which can be loaded by a processor to execute the steps of any method provided in the embodiment of the present application.

[0118] The storage medium may be a computer-readable storage medium, and the storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] Since the computer program stored in the storage medium can execute the steps of any method provided in the embodiments of the present application, the beneficial effects that can be achieved by the method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0120] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0121] It should be understood that the present application is not limited to the embodiments that have been described above and shown in the accompanying drawings, but various modifications and changes may be made without departing from the scope thereof.

Claims

1. An image processing method, wherein, Including: Performing image detection on multiple images to be processed to obtain image quality information and eye-opening state information corresponding to each of the images to be processed; Selecting, from the multiple images to be processed, an image to be processed whose image quality information meets a predetermined quality condition to obtain a base image; Selecting, from the multiple images to be processed, an image to be processed whose eye-opening state information matches that of the base image to obtain an eye image; Based on the base image and the eye image, performing multi-frame fusion processing on the multiple images to be processed to obtain a processed image.

2. The method according to claim 1, wherein, The step of, based on the base image and the eye image, performing multi-frame fusion processing on the multiple images to be processed to obtain a processed image includes: If the base image and the eye image are not the same image, using the base image as a base frame to perform multi-frame noise reduction synthesis on the multiple images to be processed to obtain a first fusion noise reduction image; Fusing the eye region in the eye image into the first fusion noise reduction image to obtain the processed image.

3. The method according to claim 1, wherein The eye-opening state information is information used to describe whether the eyes are in an open state; The step of selecting, from the multiple images to be processed, an image to be processed whose eye-opening state information matches that of the base image to obtain an eye image includes: According to the eye-opening state information corresponding to each of the images to be processed, selecting, from the multiple images to be processed, an image to be processed in which the eyes are in an open state; Selecting, from the images to be processed in which the eyes are in an open state, the one with the highest image quality to obtain the eye image.

4. The method according to claim 1, wherein, The eye-opening state information is information used to describe the degree of eye opening and the eye-opening posture; The step of selecting, from the multiple images to be processed, an image to be processed whose eye-opening state information matches that of the base image to obtain an eye image includes: Performing object feature extraction processing on the base image to obtain object feature data; According to the object feature data, obtaining object state information, where the object state information is information used to describe the object posture of the target object; Performing matching analysis on the eye-opening state information of each image to be processed and the object state information to obtain the eye image, where the eye image is the image to be processed corresponding to the eye-opening state information that matches the object state information.

5. The method according to claim 4, wherein, The step of performing matching analysis on the eye-opening state information of each image to be processed and the object state information to obtain the eye image includes: Performing state combination scoring processing on the eye-opening state information of each image to be processed and the object state information respectively to obtain a state fusion score corresponding to each image to be processed, where the state fusion score is the state fusion score of the eyes in the image to be processed and the target object in the base image; Selecting an image to be processed according to the state fusion score corresponding to each image to be processed to obtain the eye image.

6. The method according to claim 5, wherein, The object state information includes the facial state information and body state information of the target object; The step of performing state combination scoring processing on the eye-opening state information of each image to be processed and the object state information respectively to obtain a state fusion score corresponding to each image to be processed includes: The eye-opening state information of each image to be processed is respectively combined with the face state information for state combination scoring to obtain a first score corresponding to each image to be processed; The eye-opening state information of each image to be processed is respectively combined with the body state information for state combination scoring to obtain a second score corresponding to each image to be processed; Based on the first score and the second score corresponding to each image to be processed, a state fusion score corresponding to each image to be processed is obtained.

7. The method according to claim 6, wherein, The step of obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed includes: Based on the object state information, a weighting coefficient matching the object state information is obtained; Based on the weighting coefficient, the first score and the second score corresponding to each image to be processed are respectively weighted and summed to obtain a state fusion score corresponding to each image to be processed.

8. The method according to claim 1, wherein The step of performing multi-frame fusion processing on the multiple images to be processed based on the base image and the eye image to obtain a processed image includes: If the base image and the eye image are the same image, using the base image as a base frame to perform multi-frame noise reduction synthesis on the multiple images to be processed to obtain a second fusion noise reduction image; Based on the second fusion noise reduction image, the processed image is obtained.

9. The method according to claim 1, wherein The step of selecting an image to be processed with image quality information meeting a predetermined quality condition from the multiple images to be processed to obtain a base image includes one of the following methods: Based on the image quality information in each image to be processed, selecting an image to be processed with the highest image quality from the multiple images to be processed to obtain the base image; Based on the image quality information in each image to be processed, selecting one or more images to be processed with image quality higher than a predetermined quality threshold from the multiple images to be processed to obtain the base image.

10. The method according to claim 1, wherein, The image quality information includes, but is not limited to, one or more of image sharpness, resolution, contrast, brightness, noise level, color balance, image distortion degree, and aberration degree.

11. The method according to claim 6, wherein, The step of obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed includes: Adding or multiplying the first score and the second score corresponding to each image to be processed to obtain a state fusion score corresponding to each image to be processed.

12. The method according to claim 6, wherein, The step of obtaining a state fusion score corresponding to each image to be processed based on the first score and the second score corresponding to each image to be processed includes: Obtaining a unified weighting coefficient; Based on the unified weighting coefficient, the first score and the second score corresponding to each image to be processed are respectively weighted and summed to obtain a state fusion score corresponding to each image to be processed.

13. The method according to claim 5, wherein, The object state information includes the face state information of the target object; the step of respectively performing state combination scoring processing on the eye-opening state information of each image to be processed and the object state information to obtain a state fusion score corresponding to each image to be processed includes: The eye-opening state information of each image to be processed is respectively combined with the face state information for state combination scoring to obtain a first score corresponding to each image to be processed; Based on the first score corresponding to each image to be processed, obtain the state fusion score corresponding to each image to be processed.

14. The method according to claim 5, wherein, The state combination scoring process is respectively performed on the eye-opening state information of each image to be processed and the object state information to obtain the state fusion score corresponding to each image to be processed, including: Using a preset state fusion scoring neural network, perform a state combination scoring process on the eye-opening state information and the object state information of each image to be processed respectively to obtain the state fusion score corresponding to each image to be processed.

15. An image processing apparatus, wherein, Including: A detection module for performing image detection on multiple images to be processed to obtain the image quality information and the eye-opening state information corresponding to each image to be processed; An image quality information and the eye-opening state information; A selection module for selecting, from the multiple images to be processed, the images to be processed whose image quality information meets the predetermined quality conditions to obtain the base images; A matching module for selecting, from the multiple images to be processed, an image to be processed whose eye-opening state information matches the base image to obtain the eye image; A fusion module for performing multi-frame fusion processing on the multiple images to be processed based on the base image and the eye image to obtain the processed image.

16. The apparatus according to claim 15, wherein, The fusion module is used for: if the base image and the eye image are not the same image, perform multi-frame noise reduction synthesis on the multiple images to be processed with the base image as the base frame to obtain the first fusion noise reduction image; fuse the eye region in the eye image into the first fusion noise reduction image to obtain the processed image.

17. The apparatus according to claim 15, wherein, The eye-opening state information is the information used to describe whether the eyes are in an open state; the matching module is used for: according to the eye-opening state information corresponding to each image to be processed, select, from the multiple images to be processed, the images to be processed with the eyes in an open state; select the one with the highest image quality from the images to be processed with the eyes in an open state to obtain the eye image.

18. The device according to claim 15, wherein The eye-opening state information is the information used to describe the degree of eye opening and the eye-opening posture; the matching module is used for: performing object feature extraction processing on the base image to obtain object feature data; obtaining object state information according to the object feature data, where the object state information is the information used to describe the object posture of the target object; performing matching analysis on the eye-opening state information of each image to be processed and the object state information to obtain the eye image, where the eye image is the image to be processed corresponding to the eye-opening state information that matches the object state information.

19. A storage medium, wherein, Stored thereon is a computer program, which, when executed by a processor of a computer, causes the computer to execute the method according to any one of claims 1 to 14.

20. An electronic device, wherein, Including: A memory storing a computer program; A processor for reading the computer program stored in the memory to execute the method according to any one of claims 1 to 14.

Images