Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention relates to the field of image processing, and discloses an image processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a stored first facial feature, the first facial feature being a facial feature corresponding to a target user; image frames in the first video data are matched with the first facial features; and if a target image frame in the first video data comprises a first facial region matched with the first facial feature, performing first optimization processing on the target image frame according to the first facial feature to obtain an optimized target image frame. According to the embodiment of the invention, the image frame can be optimized based on the first facial feature stored in advance, and the optimization effect of the image frame is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular, to an image processing method and apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of image processing technology and the improvement of the computing power of electronic devices, users can now optimize the captured images (e.g., beauty processing, background blurring processing) through electronic devices to improve the quality of the images.

[0003] As users' requirements for image optimization effects are getting higher and higher, how to improve the optimization effect of images is an ever-optimizing goal in this field. Summary of the Invention

[0004] Embodiments of this application disclose an image processing method and apparatus, an electronic device, and a computer-readable storage medium, which can optimize image frames in video data based on pre-stored first facial features, thereby improving the image optimization effect.

[0005] In a first aspect, an embodiment of this application discloses an image processing method, which includes:

[0006] Obtain the stored first facial features, where the first facial features are the facial features corresponding to the target user;

[0007] Match the image frames in the first video data with the first facial features;

[0008] If the target image frame in the first video data includes a first facial area that matches the first facial features, perform a first optimization process on the target image frame according to the first facial features to obtain an optimized target image frame.

[0009] In a second aspect, an embodiment of this application discloses an image processing apparatus, which includes:

[0010] An obtaining unit, configured to obtain the stored first facial features, where the first facial features are the facial features corresponding to the target user;

[0011] A matching unit, configured to match the image frames in the first video data with the first facial features;

[0012] A first optimization unit, configured to, when the target image frame in the first video data includes a first facial area that matches the first facial features, perform a first optimization process on the target image frame according to the first facial features to obtain an optimized target image frame.

[0013] In a third aspect, an embodiment of this application discloses an electronic device, including:

[0014] A memory storing executable program code;

[0015] A processor coupled to the memory;

[0016] The processor calls the executable program code stored in the memory and executes the image processing method disclosed in the first aspect of the embodiments of the present application.

[0017] The fourth aspect of the embodiments of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the image processing method disclosed in the first aspect of the embodiments of the present application.

[0018] The fifth aspect of the embodiments of the present application discloses a computer program product, which, when running on a computer, causes the computer to execute some or all of the steps of any one of the methods disclosed in the first aspect of the embodiments of the present application.

[0019] The sixth aspect of the embodiments of the present application discloses an application publishing platform for publishing a computer program product, wherein, when the computer program product runs on a computer, it causes the computer to execute some or all of the steps of any one of the methods disclosed in the first aspect of the embodiments of the present application.

[0020] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0021] In the embodiments of the present application, the first facial features of the target user stored in advance can be obtained; then, the image frames in the first video data are matched with the first facial features; if the target image frame in the first video data includes a first facial area that matches the first facial features, the target image frame can be subjected to a first optimization process according to the first facial features to obtain an optimized target image frame. It should be noted that the target image frame is a frame image in the first video data. Due to various influencing factors during the shooting process, there may be situations such as blurred faces and missing facial features, which will affect the optimization effect of the target image frame; however, since the first facial features of the target user are collected and stored in advance in the embodiments of the present application, the first facial features can provide more complete and accurate facial features for achieving better optimization processing, thereby improving the optimization effect of the target image frame in the first video data; in addition, in the embodiments of the present application, when a target image frame that matches the first facial features of the target user is identified in the first video data, the target image frame including the target user can be optimized without manual triggering by the user, thereby improving the efficiency of image optimization. Description of the Drawings

[0022] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0023] Figure 1 is a schematic flowchart of an image processing method disclosed in an embodiment of the present application;

[0024] Figure 2 is a schematic flowchart of another image processing method disclosed in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a display interface disclosed in an embodiment of the present application;

[0026] Figure 4A is a schematic flowchart of yet another image processing method disclosed in an embodiment of the present application;

[0027] Figure 4B is a schematic flowchart of a process for generating a live image disclosed in an embodiment of the present application;

[0028] Figure 5 is a schematic flowchart of still another image processing method disclosed in an embodiment of the present application;

[0029] Figure 6 is a schematic structural diagram of an image processing apparatus disclosed in an embodiment of the present application;

[0030] Figure 7 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. Detailed implementation manners

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of them. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", "third", "fourth", etc. in the description and claims of this application are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "having" in the embodiments of this application and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0033] The embodiments of this application disclose an image processing method, apparatus, electronic device, and computer-readable storage medium, which can optimize an image frame based on a stored first facial feature, improving the optimization effect of the image frame.

[0034] To more clearly introduce the image processing method disclosed in the embodiments of this application, the image optimization technology in related technologies will be introduced first.

[0035] In related technologies, it is usually necessary for the user to manually trigger the image optimization process and manually set the optimization parameters, and the operation process is cumbersome, resulting in low efficiency of image optimization processing. Exemplarily, in current digital photography technology, Live Photo is an innovative form that combines static pictures and short videos. According to the existing technology, when the user takes a Live Photo, the device will automatically record the static image and the short video clips before and after. Then, when the user browses the Live Photo, they can view the dynamic effect by touching the screen. For Live Photos, the user can manually perform optimization processing through traditional image optimization methods. However, since there are a relatively large number of image frames in the static picture plus the short video of the Live Photo, the processing efficiency using traditional image optimization methods is low, reducing the user experience.

[0036] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an image processing method disclosed in the embodiments of this application.

[0037] Among them:

[0038] 102. Collect the facial image of the target user;

[0039] 104. Perform feature recognition on the facial image to obtain the first facial feature;

[0040] 106. Generate a feature file according to the first facial feature;

[0041] 108. Extract the first video data from the Live Photo;

[0042] 110. Obtain the first facial feature of the target user from the feature file; and match the image frames in the first video data with the first facial feature.

[0043] 112. If the target image frame in the first video data includes a first facial area that matches the first facial feature, the target image frame can be subjected to a first optimization process according to the first facial feature to obtain an optimized target image frame.

[0044] 114. Display a recommended cover image and the original cover image corresponding to the first video data, where the recommended cover image is any frame in the second video data.

[0045] 116. In response to a triggered cover selection operation, determine a target cover image from the recommended cover image and the original cover image.

[0046] 118. Generate a live image according to the second video data and the target cover image.

[0047] 120. Store and output the live image.

[0048] It should be noted that the target image frame is a frame image in the first video data. Due to various influencing factors during the shooting process, there may be situations such as blurred faces or missing facial features, which will affect the optimization effect of the target image frame. However, since the first facial feature of the target user is collected and stored in advance in the embodiments of the present application, the first facial feature can provide a more complete and accurate facial feature for achieving a better optimization process, thereby improving the priority effect of the target image frame in the first video data. In addition, in the embodiments of the present application, when a target image frame that matches the first facial feature of the target user is recognized in the first video data, the target image frame including the target user can be optimized without manual triggering by the user, thereby improving the efficiency of image optimization.

[0049] Based on this, the following introduces the image processing method and apparatus, electronic device, and computer-readable storage medium disclosed in the embodiments of the present application.

[0050] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another image processing method disclosed in the embodiments of the present application. Optionally, this method can be applied to various electronic devices with image processing capabilities, or other execution entities (such as cloud servers, local servers, etc., which are not limited here), and are not limited here. Optionally, this method can include the following steps:

[0051] 202. Obtain the stored first facial feature, where the first facial feature is the facial feature corresponding to the target user.

[0052] In an embodiment of the present application, an electronic device may store a first facial feature corresponding to a target user. The first facial feature may be used to represent the facial information of the target user and is used to distinguish the face of the target user from the faces of other users.

[0053] Optionally, the first facial feature may include one or more of: facial contour features, the position and proportion adjustment of facial features, and skin color, texture, and flaw features, which are not limited herein. Optionally, the facial contour features may include one or more of: chin contour features, forehead contour features, and cheek contour features, which are not limited herein.

[0054] Optionally, the number of corresponding features of the first facial feature is greater than or equal to a number threshold; wherein, the number threshold may be set by developers based on a large amount of development experience and is not limited herein, so that the first facial feature can be more complete and accurate.

[0055] Optionally, the electronic device may collect a face image corresponding to the target user; perform feature recognition on the face image to obtain the first facial feature, and then generate a feature file according to the first facial feature.

[0056] By implementing the above method, the electronic device can collect and store the first facial feature of the target user in advance to provide a data basis for subsequent image optimization processes, thereby improving the effect of subsequent image optimization of the target image frame according to the first facial feature.

[0057] Optionally, the electronic device may collect a face image of the target user through a camera; wherein, the camera may include a front camera. Please refer to Figure 3 , Figure 3 which is a schematic diagram of a display interface disclosed in an embodiment of the present application. Optionally, the electronic device may output a first prompt message 320 on the display interface 310. The first prompt message 320 is used to prompt the target user to place the face at a target position (for example: within the square 330 output by the display interface) to collect the face image of the user.

[0058] Optionally, when the electronic device detects the face of the target user, it may output a second prompt message, which is used to prompt the target user to turn the head so that the camera can collect more frames of face images at different angles, so that more first facial features can be collected subsequently.

[0059] Optionally, the electronic device may perform feature recognition on the face image through a face recognition algorithm to obtain the first facial feature. Optionally, the face recognition algorithm may include: a face key point detection algorithm, a Dlib algorithm, or an MTCNN (Multi-Task Cascaded Convolutional Networks) algorithm, which are not limited herein.

[0060] Optionally, the electronic device may graphically mark the first facial feature and generate a feature file based on the first facial feature. Optionally, the format of the feature file may include the JSON format, or the XML format, which is not limited herein.

[0061] Optionally, before collecting the facial image corresponding to the target user, the electronic device may output a third prompt message, which is used to prompt the user whether to agree to collect the first facial feature and whether to agree to store the feature file; in response to the triggered operation of agreeing to collect, the electronic device may perform the step of collecting the facial image corresponding to the target user.

[0062] Implementing the above method can protect the privacy of users, thereby improving the user experience.

[0063] Optionally, when the electronic device needs to obtain the first facial feature, it may parse the corresponding feature file to obtain the first facial feature.

[0064] Optionally, the electronic device may store feature files corresponding to multiple users respectively, and each feature file may be bound to the identity recognition information of the corresponding user; optionally, before parsing the corresponding feature file to obtain the first facial feature, the electronic device may determine the corresponding feature file in response to the input identity recognition information. Optionally, the identity recognition information may include: name, fingerprint information, voiceprint information, etc., which is not limited herein.

[0065] Implementing the above method, the electronic device can first determine the user currently using the electronic device through the identity recognition information, and then determine the corresponding feature file, thereby improving the intelligence level of the method, and also enabling the subsequent image optimization process to target the user currently using the electronic device, improving the user experience.

[0066] In some other optional embodiments, the first facial feature may be obtained from other devices; or extracted from historical images including the target user stored in the past, which is not limited herein.

[0067] Optionally, when the electronic device collects the first facial feature of the target user, it may determine whether the number of features of the first facial feature is greater than or equal to a number threshold; if so, generate a feature file based on the first facial feature, thereby ensuring the integrity of the first facial feature, providing more data basis for subsequent image optimization, and improving the optimization effect.

[0068] In an alternative embodiment, the first facial feature may include: at least two sub-facial features, where the at least two sub-facial features are facial features collected at different shooting distances; that is, for the facial area of the same target user, facial images can be collected at different distances from the target user respectively, and then the facial features corresponding to different shooting distances are recognized as sub-facial features.

[0069] It can be understood that the target user in the target image frame may be photographed at positions with different distances from the camera, and different shooting distances will result in different sizes and details of the facial area of the target user in the target image frame. For this, facial features corresponding to at least two different shooting distances can be stored to improve the success rate of subsequent matching of the image frames in the first video data with the first facial features.

[0070] Optionally, the electronic device can match the image frames in the first video data with the first facial features. If the target image frame in the first video data includes a first facial area that matches the first sub-facial feature in the first facial features, it is determined whether there is any other sub-facial feature in the sub-facial features included in the first facial feature whose corresponding feature quantity is greater than that of the first sub-facial feature;

[0071] If there is, the sub-facial feature with the largest feature quantity among the other sub-facial features whose feature quantity is greater than that of the first sub-facial feature is used as the target sub-facial feature, and the target image frame is subjected to a first optimization process according to the target sub-facial feature to obtain an optimized target image frame.

[0072] It should be noted that the first sub-facial feature that is matched may be a facial feature collected when the target user is at a relatively far distance from the camera, and the feature quantity is small. In this case, the amount of data available for image optimization is small, which is less helpful for improving the image optimization effect. In the embodiment of the present application, facial features corresponding to at least two different shooting distances of the same target user are stored, and the feature quantity of the target sub-facial feature collected at a relatively close shooting distance is usually large. In this case, the target image frame can be optimized according to the target sub-facial feature with more feature quantity to improve the image optimization effect.

[0073] 204. Match the image frames in the first video data with the first facial features.

[0074] In the embodiment of the present application, the first video data may include: video data collected by the electronic device, video data downloaded from the Internet, or short video data included in the live image, which is not limited herein.

[0075] The first video data includes multiple image frames, and the electronic device can sequentially match the image frames in the first video data with the first facial feature. Optionally, the electronic device can identify whether the image frame includes a facial region. If it includes a facial region, the third facial feature corresponding to the facial region is extracted, and the third facial feature is matched with the first facial feature.

[0076] If the third facial feature matches the first facial feature, it is determined that the image frame includes the first facial region that matches the first facial feature; if the third facial feature does not match the first facial feature, it is determined that the image frame does not include the first facial region that matches the first facial feature.

[0077] 206. If the target image frame in the first video data includes the first facial region that matches the first facial feature, the target image frame is subjected to a first optimization process according to the first facial feature to obtain an optimized target image frame.

[0078] In the embodiment of the present application, the target image frame may be an image frame in the first video data that includes the first facial region that matches the first facial feature. The target image frame includes the target user. In this regard, the electronic device can perform a first optimization process on the target image frame according to the first facial feature to obtain an optimized target image frame. Optionally, the first optimization process may include one or more of image restoration, eyebrow refinement, skin smoothing, and tooth brightness adjustment, which is not limited herein.

[0079] In another optional embodiment, if the image frame in the first video data does not include the first facial region that matches the first facial feature, a third optimization operation may be performed on the image frame, or no optimization operation may be performed, which is not limited herein.

[0080] Optionally, the third optimization operation may include one or more of blurring the background region, filling in details of the background region, and enhancing the image quality, which is not limited herein.

[0081] In an optional embodiment, the electronic device can perform a fourth optimization operation on each image frame in the first video data through a pre-trained image optimization model. The fourth optimization operation is a basic optimization operation and / or a special effect processing operation, which is not limited herein.

[0082] In an optional embodiment, the electronic device can identify the target facial feature in the first facial region of the target image frame, and the target facial feature is a feature that can characterize the uniqueness of the target user.

[0083] Optionally, the target facial features may include one or more of eyebrows and eyes, facial contour, lip contour, nose contour, and ear contour, which are not limited herein. Optionally, the electronic device may perform an optimization operation on the target facial features corresponding to the target image frame according to the first facial feature.

[0084] By implementing the above method, the electronic device can perform an optimization operation on the target facial features that can represent the uniqueness of the target user in the target image frame, so that it is easier to recognize the face of the target user in the optimized target image frame.

[0085] By implementing the methods disclosed in the above embodiments, the first facial feature of the target user stored in advance can be obtained; then, the image frame in the first video data is matched with the first facial feature; if the target image frame in the first video data includes a first facial area that matches the first facial feature, the target image frame can be subjected to a first optimization process according to the first facial feature to obtain an optimized target image frame. It should be noted that the target image frame is a frame image in the first video data. Due to various influencing factors during the shooting process, there may be situations such as blurred face and missing facial features, which will affect the optimization effect of the target image frame; however, since the first facial feature of the target user is collected and stored in advance in the embodiments of the present application, the first facial feature can provide a more complete and accurate facial feature for implementing a better optimization process, thereby improving the optimization effect on the target image frame in the first video data; in addition, in the embodiments of the present application, when a target image frame that matches the first facial feature of the target user is recognized in the first video data, the target image frame including the target user can be optimized without manual triggering by the user, thereby improving the efficiency of image optimization.

[0086] Please refer to Figure 4A , Figure 4A which is a schematic flowchart of another image processing method disclosed in the embodiments of the present application. Optionally, this method can be applied to various electronic devices with image processing capabilities, or other execution entities (such as cloud servers, local servers, etc., which are not limited herein), which are not limited herein. Optionally, this method may include the following steps:

[0087] 402. Obtain the stored first facial feature, where the first facial feature is the facial feature corresponding to the target user.

[0088] 404. Match the image frame in the first video data with the first facial feature.

[0089] 406. If the target image frame in the first video data includes a first facial area that matches the first facial feature, perform a first optimization process on the target image frame according to the first facial feature to obtain an optimized target image frame.

[0090] As an alternative embodiment, the first optimization process may include: image reconstruction processing. Image reconstruction processing refers to the process of recovering a clear and complete image from incomplete, damaged, or low-quality image data; its core objective is to restore or enhance image information to make it closer to the original scene or meet specific requirements.

[0091] It should be noted that in the related art, the missing details in the image are usually reconstructed through a learning model or a prediction algorithm. However, although the prediction results in the related art can restore the image to a certain extent, the effect is not very satisfactory.

[0092] In the embodiment of the present application, the electronic device has pre-stored the first facial feature of the target user. Therefore, the electronic device has a data basis for repairing the data in the first facial area of the target image frame. Therefore, the electronic device can perform image reconstruction processing on the first facial area in the target image frame according to the first facial feature to improve the effect of image reconstruction.

[0093] In an alternative embodiment, the electronic device may obtain a second facial feature corresponding to the first facial area; and perform image reconstruction processing on the first facial area in the target image frame according to the first facial feature and the second facial feature.

[0094] In an alternative embodiment, the image reconstruction processing may include: resolution reconstruction processing. Resolution reconstruction refers to the process of enhancing the clarity and details of an image through an algorithm and converting a low-resolution image into a high-resolution image.

[0095] Optionally, the electronic device may determine the element parameters corresponding to the reconstruction elements according to the first facial feature and the second facial feature; wherein, the reconstruction elements may include pixel points; furthermore, the electronic device may perform resolution reconstruction processing on the first facial area in the target image frame according to the element parameters corresponding to the pixel points.

[0096] Optionally, the electronic device may determine the element parameters corresponding to the reconstruction elements according to the first facial feature and the second facial feature through a super-resolution reconstruction algorithm. Among them, the super-resolution reconstruction algorithm may include: Super-Resolution Convolutional Neural Network (SRCNN), Enhanced Super-Resolution GAN (ESRGAN), etc., which are not limited herein.

[0097] Implementing the above method, the electronic device can perform resolution reconstruction processing on the first facial region in the target image frame according to the first facial feature to improve the resolution of the first facial region, thereby improving the image quality of the target image frame and enabling the target image frame to still retain a natural effect after being enlarged.

[0098] In another alternative embodiment, the image reconstruction processing may include: texture feature reconstruction processing. Texture feature reconstruction refers to restoring the texture features and details of an image through an algorithm.

[0099] Optionally, the electronic device can determine the element parameters corresponding to the reconstruction elements according to the first facial feature and the second facial feature; wherein, the reconstruction elements may include texture features and facial features; furthermore, the electronic device can perform texture feature reconstruction processing on the first facial region in the target image frame according to the element parameters.

[0100] Optionally, the electronic device can determine the element parameters corresponding to the reconstruction elements according to the first facial feature and the second facial feature through an image inpainting algorithm. The image inpainting algorithm may include: an image inpainting method based on a convolutional neural network (CNN), an image inpainting method based on a generative adversarial network (GAN), etc., which are not limited herein.

[0101] Implementing the above method, the electronic device can reconstruct the missing or damaged part in the target image frame based on the first facial feature, and in the case of having the first facial feature, can more accurately predict and reconstruct the missing facial features, making the image more complete and natural, and improving the optimization effect on the target image frame.

[0102] In one alternative embodiment, the first optimization processing may include style transfer processing. Among them, style transfer processing refers to applying the artistic style of one image (such as oil painting brushstrokes, color matching, texture effects) to the content of another image to generate a new image that retains the original image content and has a new style.

[0103] Optionally, the electronic device can determine the facial structure information of the target user according to the first facial feature, and the facial structure information includes the proportion information and / or position information of the facial features of the target user; furthermore, the electronic device can perform style transfer processing on the target image frame according to the facial structure information.

[0104] Optionally, the style transfer processing can be implemented through a generative adversarial network or a neural style transfer algorithm, which is not limited herein.

[0105] It should be noted that when performing style transfer processing, it is necessary to preserve the content of the original image as much as possible. In the embodiments of the present application, the electronic device has obtained the complete and accurate first facial feature in advance. To implement the above method, the electronic device can determine the facial structure information of the target user according to the first facial feature, and then, while preserving the facial structure of the target user as much as possible, perform style transfer processing, thereby improving the effect of style transfer processing.

[0106] In an alternative embodiment, the first optimization process includes beauty treatment, which refers to automatically modifying the skin, facial features, and contours in portrait photos or videos through image algorithms to make them look smoother, more delicate, or conform to aesthetic standards.

[0107] In practice, it is found that there may be cases of makeup smudging during shooting, resulting in the loss of makeup in the facial image frames. For this, some users have a need to touch up the makeup in the facial area of the image frames. Optionally, the beauty treatment in the embodiments of the present application may include makeup beautification treatment.

[0108] Optionally, the electronic device can determine the makeup beauty parameters of the target user according to the first facial feature; and then, according to the makeup beauty parameters, perform beauty treatment on the first facial area in the target image frame. Among them, the makeup beauty parameters are used to characterize the makeup style of the target user.

[0109] Implementing the above method, the electronic device can determine the daily makeup style of the target user according to the first facial feature to obtain the corresponding makeup beauty parameters; and then the electronic device can perform targeted and personalized beauty treatment on the first facial area of the target user in the target image frame according to the makeup beauty parameters, so that the beauty treatment is more matched with the daily makeup style of the target user, improving the user experience.

[0110] 408. Generate second video data according to the optimized multi-frame image frames.

[0111] In the embodiments of the present application, the optimized multi-frame image frames may include: the optimized target image frame and other image frames in the first video data except the target image frame, which is not limited herein.

[0112] Optionally, the electronic device can display the second video data for the user to refer to; further, in response to a target adjustment operation on the second video data, the electronic device can perform a target adjustment operation on the second video data according to the target adjustment operation. The target adjustment operation may include: adjusting image attribute parameters, editing operations, etc., which is not limited herein.

[0113] 410. Generate a live image according to the second video data and the target cover image.

[0114] As introduced above, the live image is an innovative form that combines static pictures and short videos. How to generate a live image based on the second video data and the target cover image is the prior art and will not be elaborated here.

[0115] By implementing the above method, the electronic device can generate the second video data based on the optimized image frames, and generate a live image with better picture quality based on the second video data and the target cover image, thereby improving the image quality of the live image.

[0116] As an optional implementation manner, the first video data may be the video data included in the original live image, and the original live image further includes an original cover image.

[0117] Optionally, before generating a live image based on the second video data and the target cover image, the electronic device may display the recommended cover image and the original cover image corresponding to the first video data for the user to refer to; wherein, the recommended cover image is any frame in the second video data.

[0118] Optionally, the electronic device may determine the image frame with the best optimized quality in the second video data as the recommended cover image. In another optional embodiment, the electronic device may determine the image frame that can represent the central idea of the second video data in the second video data as the recommended cover image, which is not limited herein.

[0119] Furthermore, the user may select any frame from the displayed recommended cover image and the original cover image corresponding to the first video data as the facial cover image. In response to the triggered cover selection operation, the electronic device may determine the image selected by the cover selection operation in the recommended cover image and the original cover image as the target cover image.

[0120] By implementing the above method, the electronic device can display the optimized recommended cover image and the original cover image for the user to refer to and select, so as to determine the cover image desired by the user, thereby improving the controllability of the method and enhancing the user experience.

[0121] Exemplarily, please refer to Figure 4B , Figure 4B is a schematic flowchart of a process for generating a live image disclosed in an embodiment of the present application. Wherein:

[0122] The electronic device may select a frame of recommended cover image 414 from the second video data 412, and display the recommended cover image 414 and the original cover image 416 corresponding to the first video data together for the user to refer to;

[0123] Furthermore, in response to the cover selection operation, determine a frame in the recommended cover image 414 and the original cover image 416 as the target cover image 418;

[0124] The electronic device may then generate a live image 420 based on the target cover image 418 and the second video data 412 .

[0125] By implementing the methods disclosed in the above embodiments, the first facial features of the target user are collected and stored in advance. The first facial features can provide more complete and accurate facial features for achieving better optimization processing, thereby improving the priority effect on the target image frame in the first video data; in addition, in the embodiment of the present application, when a target image frame matching the first facial feature of the target user is identified in the first video data, the target image frame including the target user can be optimized without the need for manual triggering by the user, thereby improving the efficiency of image optimization; and, the first facial area in the target image frame can be image reconstructed according to the first facial feature to improve the effect of image reconstruction; and, the first facial area in the target image frame can be resolution reconstructed according to the first facial feature to improve the resolution of the first facial area;

[0126] Furthermore, the missing or damaged part of the target image frame can be reconstructed based on the first facial feature, and in the case of the first facial feature, the missing facial feature can be more accurately predicted and reconstructed, making the image more complete and natural; further, the facial structure information of the target user can be determined based on the first facial feature, and then style transfer processing can be performed while retaining the facial structure of the target user as much as possible, thereby improving the effect of the style transfer processing; further, the first facial area of the target user in the target image frame can be subjected to targeted and personalized beauty processing based on makeup beauty parameters, thereby making the beauty processing more compatible with the daily makeup style of the target user, thereby improving the user experience; further, the optimized recommended cover image and the original cover image can be displayed for the user's reference and selection, so as to determine the cover image desired by the user, thereby improving the controllability of the method and improving the user experience.

[0127] See also Figure 5 , Figure 5 : is a flowchart of another image processing method disclosed in an embodiment of the present application. Optionally, the method can be applied to various electronic devices with image processing capabilities, or other execution entities (for example, cloud servers, local servers, etc., which are not limited here). Optionally, the method can include the following steps:

[0128] 502. Obtain a stored first facial feature, where the first facial feature is a facial feature corresponding to a target user.

[0129] 504. Match the image frame in the first video data with the first facial feature.

[0130] 506. If the target image frame in the first video data includes a first facial area that matches the first facial feature, the target image frame is subjected to a first optimization process according to the first facial feature to obtain an optimized target image frame.

[0131] As an optional implementation, the electronic device can recognize the first expression information corresponding to the first facial area of the target image frame and determine the expected expression information according to the first expression information.

[0132] Optionally, the electronic device can use an expression recognition model to recognize the first expression information and determine the expected expression information, which is not limited herein. Optionally, the expression recognition model can include: a trained CNN model (such as VGG-Face, ResNet) or a dedicated expression recognition model (such as FER2013, AffectNet), etc., which is not limited herein.

[0133] Wherein, the first expression information is the real expression of the target user in the target image frame (for example: smiling), and the expected expression information is the expected modified expression (for example: laughing), which is not limited herein.

[0134] Further, after the target image frame is subjected to a first optimization process according to the first facial feature to obtain an optimized target image frame, the electronic device can perform an expression adjustment process on the first facial area in the optimized target image frame according to the expected expression information.

[0135] Implementing the above method, since the first optimization process is performed on the target image frame using the first facial feature information, the feature information of the first facial area in the optimized target image frame is more abundant. Therefore, based on the optimized target image frame, a more natural facial expression can be generated or adjusted, enhancing the emotional communication strength and thus improving the image optimization effect.

[0136] As an optional implementation, the electronic device can recognize the information of the key points of the facial features in the first facial area of the target image frame; wherein, the information of the key points of the facial features can include: such as the edges of the eyes, nose, lips, etc., which is not limited herein.

[0137] Furthermore, the electronic device can perform a target adjustment operation on the facial features in the first facial area according to the information of the key points of the facial features. Optionally, the target adjustment operation includes: enhancing the facial feature and / or smoothing the facial feature, etc., which is not limited herein.

[0138] Optionally, the electronic device can use a facial feature point recognition algorithm to recognize the information of the key points of the facial features in the first facial area of the target image frame. Optionally, the facial feature point recognition algorithm can include the Dlib algorithm or the MTCNN algorithm introduced above, which is not limited herein.

[0139] By implementing the above method, the electronic device can identify the facial feature key points information in the target image frame, and perform processing such as feature enhancement and / or feature smoothing on the facial features based on the facial feature key points information, so as to improve the overall facial quality and natural beauty, thereby improving the image optimization effect.

[0140] 508. Perform a second optimization operation on other regions in the target image frame, where the other regions are the regions in the target image frame except the first facial region.

[0141] In the embodiment of the present application, the second optimization operation may include: performing beauty treatment on the second facial region, performing blurring treatment on the background region, performing detail completion treatment on the background region, and performing picture quality enhancement treatment, etc., where the second facial region is the facial region corresponding to other users.

[0142] Optionally, the electronic device may also perform a facial highlighting adjustment operation on the second facial region. Optionally, the facial highlighting adjustment operation may include: adjusting (such as increasing) light, contrast, saturation, etc., which is not limited herein.

[0143] By implementing the above method, the electronic device can perform a second optimization operation on other regions in the target image frame to improve the overall priority effect of the target image frame.

[0144] By implementing the methods disclosed in the above embodiments, the first facial features of the target user are collected and stored in advance. The first facial features can provide more complete and accurate facial features for implementing better optimization processing, thereby improving the priority effect of the target image frame in the first video data; in addition, in the embodiment of the present application, when a target image frame matching the first facial features of the target user is identified in the first video data, optimization processing can be performed on the target image frame including the target user without manual triggering by the user, thereby improving the efficiency of image optimization; and, based on the optimized target image frame, a more natural facial expression can be generated or adjusted, enhancing the emotional communication strength, thereby improving the image optimization effect; and, the facial feature key points information in the target image frame can be identified, and feature enhancement and / or feature smoothing, etc. can be performed on the facial features based on the facial feature key points information to improve the overall facial quality and natural beauty, thereby improving the image optimization effect; and, a second optimization operation can be performed on other regions in the target image frame to improve the overall priority effect of the target image frame.

[0145] Please refer to Figure 6 , Figure 6It is a schematic structural diagram of an image processing device disclosed in an embodiment of the present application. Optionally, this method can be applied to various electronic devices with image processing capabilities, or other execution entities (such as cloud servers, local servers, etc., which are not limited here), which are not limited here. Optionally, the device may include an acquisition unit 602, a matching unit 604, and a first optimization unit 606, where:

[0146] The acquisition unit 602 is configured to acquire a stored first facial feature, where the first facial feature is a facial feature corresponding to a target user;

[0147] The matching unit 604 is configured to match an image frame in the first video data with the first facial feature;

[0148] The first optimization unit 606 is configured to, when a target image frame in the first video data includes a first facial area that matches the first facial feature, perform a first optimization process on the target image frame according to the first facial feature to obtain an optimized target image frame.

[0149] By implementing the above device, the first facial feature of the target user stored in advance can be acquired; then, the image frames in the first video data are matched with the first facial feature; if the target image frame in the first video data includes a first facial area that matches the first facial feature, the first optimization process can be performed on the target image frame according to the first facial feature to obtain an optimized target image frame. It should be noted that the target image frame is a frame image in the first video data. Due to various influencing factors during the shooting process, there may be situations such as blurred faces and missing facial features, which will affect the optimization effect of the target image frame; however, since the first facial feature of the target user is collected and stored in advance in the embodiment of the present application, the first facial feature can provide a more complete and accurate facial feature for achieving a better optimization process, thereby improving the priority effect on the target image frame in the first video data; in addition, in the embodiment of the present application, when a target image frame that matches the first facial feature of the target user is recognized in the first video data, the optimization process can be performed on the target image frame including the target user without manual triggering by the user, thereby improving the efficiency of image optimization.

[0150] As an optional implementation manner, Figure 6 The device shown may further include an acquisition unit (not shown), where:

[0151] The acquisition unit is configured to, before acquiring the stored first facial feature, acquire a face image corresponding to the target user; perform feature recognition on the face image to obtain the first facial feature, and generate a feature file according to the first facial feature;

[0152] Moreover, the obtaining unit 602 is further configured to parse the stored feature file to obtain the first facial feature.

[0153] By implementing the above device, the electronic device can collect and store the first facial feature of the target user in advance, providing a data basis for the subsequent image optimization process, thereby improving the effect of subsequent image optimization of the target image frame according to the first facial feature.

[0154] As an alternative implementation manner, Figure 6 The device shown may further include a first generating unit and a second generating unit not shown in the figure, where:

[0155] The first generating unit is configured to generate second video data according to the optimized multiple image frames after performing a first optimization process on the target image frame according to the first facial feature to obtain the optimized target image frame;

[0156] The second generating unit is configured to generate a live image according to the second video data and the target cover image.

[0157] By implementing the above device, the electronic device can generate second video data according to the optimized image frames, and generate a live image with better picture quality according to the second video data and the target cover image, thereby improving the image quality of the live image.

[0158] As an alternative implementation manner, Figure 6 The device shown may further include a determining unit not shown in the figure, where:

[0159] The determining unit is configured to display a recommended cover image and the original cover image corresponding to the first video data before generating a live image according to the second video data and the target cover image, where the recommended cover image is any frame in the second video data; in response to a triggered cover selection operation, determine the target cover image from the recommended cover image and the original cover image.

[0160] By implementing the above device, the electronic device can display the optimized recommended cover image and the original cover image for the user to refer to and select, so as to determine the cover image expected by the user, thereby improving the controllability of the method and enhancing the user experience.

[0161] As an alternative implementation manner, the first optimization process includes an image reconstruction process; the first optimization unit 606 is further configured to obtain a second facial feature corresponding to the first facial area; and perform an image reconstruction process on the first facial area in the target image frame according to the first facial feature and the second facial feature.

[0162] By implementing the above-mentioned device, since the electronic device stores the first facial features of the target user in advance, the electronic device has a data basis for repairing the first facial area in the target image frame. The electronic device can perform image reconstruction processing on the first facial area in the target image frame according to the first facial features to improve the effect of image reconstruction.

[0163] As an optional implementation, the image reconstruction process includes: resolution reconstruction processing and / or texture feature reconstruction processing; the first optimization unit 606 is further used to determine element parameters corresponding to the reconstructed elements based on the first facial features and the second facial features; and, based on the element parameters, perform resolution reconstruction processing and / or texture feature reconstruction processing on the first facial area in the target image frame.

[0164] By implementing the above device, the electronic device can reconstruct the missing or damaged part of the target image frame based on the first facial feature, and in the case of the first facial feature, the missing facial feature can be more accurately predicted and reconstructed, making the image more complete and natural, thereby improving the optimization effect of the target image frame.

[0165] As an optional implementation, the first optimization processing includes style transfer processing; the first optimization unit 606 is also used to determine the facial structure information of the target user based on the first facial feature, the facial structure information including the facial features proportion information and / or position information of the target user; and, based on the facial structure information, perform style transfer processing on the target image frame.

[0166] By implementing the above device, the electronic device can determine the facial structure information of the target user according to the first facial feature, and then perform style migration processing while retaining the facial structure of the target user as much as possible, thereby improving the effect of the style migration processing.

[0167] As an optional implementation, the first optimization processing includes beautification processing; the first optimization unit 606 is also used to determine the makeup beautification parameters of the target user based on the first facial features; and, based on the makeup beautification parameters, perform beautification processing on the first facial area in the target image frame.

[0168] By implementing the above-mentioned device, the electronic device can determine the target user's daily makeup style based on the first facial feature to obtain corresponding makeup beauty parameters; then the electronic device can perform targeted and personalized beauty processing on the first facial area of the target user in the target image frame according to the makeup beauty parameters, so that the beauty processing is more matched with the target user's daily makeup style, thereby improving the user's usage experience.

[0169] As an optional implementation, Figure 6 The device shown may further include an identification unit and a first adjustment unit (not shown), wherein:

[0170] An identification unit, configured to identify first facial expression information corresponding to a first facial area of a target image frame, and determine expected expression information according to the first facial expression information;

[0171] A first adjustment unit, configured to perform an expression adjustment process on the first facial area in the optimized target image frame according to the expected expression information after performing a first optimization process on the target image frame according to the first facial features to obtain an optimized target image frame.

[0172] Implementing the above device, since the first optimization process is performed on the target image frame using the first facial feature information, the feature information of the first facial area in the optimized target image frame is made more abundant. Thus, based on the optimized target image frame, a more natural facial expression can be generated or adjusted, enhancing the emotional communication strength, and thereby improving the image optimization effect.

[0173] As an optional implementation manner, Figure 6 The device shown may further include a second adjustment unit (not shown), where:

[0174] The second adjustment unit is configured to identify the key point information of the facial features in the first facial area of the target image frame; and perform a target adjustment operation on the facial features in the first facial area according to the key point information of the facial features, and the target adjustment operation includes: enhancing the facial feature and / or smoothing the facial feature.

[0175] Implementing the above device, the electronic device can identify the key point information of the facial features in the target image frame, and perform processing such as enhancing the facial feature and / or smoothing the facial feature on the facial features based on the key point information of the facial features, so as to improve the overall facial quality and natural beauty, thereby improving the image optimization effect.

[0176] As an optional implementation manner, Figure 6 The device shown may further include a second optimization unit (not shown), where:

[0177] The second optimization unit is configured to perform a second optimization operation on other areas in the target image frame, and the other areas are other areas in the target image frame except the first facial area;

[0178] The second optimization operation includes: performing beauty treatment on the second facial area, performing blurring treatment on the background area, performing detail completion treatment on the background area, and performing one or more of image quality enhancement treatments, and the second facial area is the facial area corresponding to other users.

[0179] Implementing the above device, the electronic device can perform a second optimization operation on other areas in the target image frame to improve the overall priority effect of the target image frame.

[0180] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. As shown in Figure 7 , the electronic device may include: a memory 701 storing executable program code; a processor 702 coupled to the memory 701;

[0181] wherein, the processor 702 calls the executable program code stored in the memory 701 and executes the image processing method disclosed in each of the above embodiments.

[0182] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the image processing method disclosed in each of the above embodiments.

[0183] An embodiment of the present application also discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product, and when the computer program product runs on a computer, it causes the computer to execute some or all of the steps of the methods in the above method embodiments.

[0184] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0185] In various embodiments of the present application, it should be understood that the magnitudes of the serial numbers of the above processes do not necessarily mean the order of execution in a necessary sequence. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0186] The units described as separate components above may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, in each embodiment of the present application, each functional unit may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0188] When the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several requests for causing a computer device (which can be a personal computer, a server, or a network device, etc., specifically, the processor in the computer device) to execute some or all of the steps of the above methods in various embodiments of the present application.

[0189] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0190] The above has introduced in detail the image processing method and apparatus, electronic device, and computer-readable storage medium disclosed in the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An image processing method, characterized in that, The method includes: Obtaining the stored first facial feature, where the first facial feature is the facial feature corresponding to the target user; Matching the image frames in the first video data with the first facial feature; If the target image frame in the first video data includes a first facial area that matches the first facial feature, then performing a first optimization process on the target image frame according to the first facial feature to obtain an optimized target image frame.

2. The method according to claim 1, wherein Before obtaining the stored first facial feature, the method further includes: Collecting a face image corresponding to the target user through a camera; Performing feature recognition on the face image to obtain a first facial feature, and generating a feature file according to the first facial feature; And, obtaining the stored first facial feature includes: Parsing the stored feature file to obtain the first facial feature.

3. The method according to claim 1, wherein After performing the first optimization process on the target image frame according to the first facial feature to obtain an optimized target image frame, the method further includes: Generating second video data according to the optimized multiple image frames; Generating a live image according to the second video data and the target cover image.

4. The method according to claim 3, wherein Before generating the live image according to the second video data and the target cover image, the method further includes: Displaying a recommended cover image and the original cover image corresponding to the first video data, where the recommended cover image is one of the frames in the second video data; In response to a triggered cover selection operation, determining the target cover image from the recommended cover image and the original cover image.

5. The method according to any one of claims 1 to 4, characterized in that The first optimization process includes image reconstruction processing; performing the first optimization process on the target image frame according to the first facial feature includes: Obtaining a second facial feature corresponding to the first facial area; Performing image reconstruction processing on the first facial area in the target image frame according to the first facial feature and the second facial feature.

6. The method according to claim 5, wherein The image reconstruction processing includes: resolution reconstruction processing and / or texture feature reconstruction processing; performing image reconstruction processing on the first facial area in the target image frame according to the first facial feature and the second facial feature includes: Determining the element parameters corresponding to the reconstruction elements according to the first facial feature and the second facial feature; Performing resolution reconstruction processing and / or texture feature reconstruction processing on the first facial area in the target image frame according to the element parameters.

7. The method according to any one of claims 1 to 4, characterized in that, The first optimization process includes style transfer processing; performing the first optimization process on the target image frame according to the first facial feature includes: Determining the facial structure information of the target user according to the first facial feature; Performing style transfer processing on the target image frame according to the facial structure information.

8. The method according to any one of claims 1 to 4, characterized in that The first optimization process includes beauty processing; performing the first optimization process on the target image frame according to the first facial feature includes: Determining the beauty parameters of the target user according to the first facial feature; Performing beauty processing on the first facial area in the target image frame according to the beauty parameters.

9. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Identify the first facial region of the target image frame corresponding to the first expression information, and determine the expected expression information according to the first expression information; And, after performing the first optimization process on the target image frame according to the first facial feature to obtain the optimized target image frame, the method further includes: Perform an expression adjustment process on the first facial region in the optimized target image frame according to the expected expression information.

10. The method according to any one of claims 1 to 4, characterized in that The method further includes: Identify the key point information of the facial features in the first facial region of the target image frame; Perform facial feature enhancement processing and / or facial feature smoothing processing on the first facial region according to the key point information of the facial features.

11. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Perform a second optimization operation on other regions in the target image frame, where the other regions are other regions in the target image frame except the first facial region; The second optimization operation includes: performing one or more of beauty processing on the second facial region, blurring processing on the background region, detail completion processing on the background region, and image quality enhancement processing, where the second facial region is the facial region corresponding to other users.

12. An image processing apparatus, characterized in that, The device includes: An acquisition unit, configured to acquire the stored first facial feature, where the first facial feature is the facial feature corresponding to the target user; A matching unit, configured to match the image frame in the first video data with the first facial feature; A first optimization unit, configured to, when the target image frame in the first video data includes a first facial region that matches the first facial feature, perform a first optimization process on the target image frame according to the first facial feature to obtain the optimized target image frame.

13. An electronic device, characterized in that, It includes a memory storing executable program code, and a processor coupled to the memory; wherein, the processor calls the executable program code stored in the memory and executes the method according to any one of claims 1 to 11.

14. A computer-readable storage medium stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 11.

Images