Image processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a storage medium, and relates to the field of image processing.The method comprises the steps that a video clip shot by a camera is acquired; wherein the face of at least one shooting object is displayed in the video clip; performing face detection and key point detection on each video frame in the video clip to obtain a face image and a face key point corresponding to each shot object; generating a first face image of each shooting object according to the face image and the face key point of each shooting object; and generating a target image according to the target video frame in the video clip and the first face image of each shooting object. Therefore, the generation quality of the target image can be improved, the shooting experience of a user is improved, the opportunity of opening eyes of all the shooting objects does not need to be waited, and the flexibility and generalization of the method can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, chip and storage medium. Background Art

[0002] As a unique and widely used identity authentication information, face has always been a research hotspot in the field of computer science. In addition to traditional research directions such as face detection, face recognition, and liveness detection, the field of face synthesis has also shown extremely high activity in recent years. With the breakthrough progress of deep learning technology, face synthesis technology is experiencing rapid development and has been widely used in many fields such as privacy protection, film and television animation, and commercial entertainment. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the present application proposes an image processing method, device, electronic device, chip and storage medium, so as to not only improve the image quality and improve the user's shooting experience, but also eliminate the need to wait for all subjects to open their eyes, thereby improving the flexibility and generalization of the method.

[0005] In one aspect, an embodiment of the present application provides an image processing method, including:

[0006] Obtain a video clip captured by a camera; wherein the video clip displays at least one face of a subject;

[0007] Performing face detection and key point detection on each video frame in the video clip to obtain a face image and face key points corresponding to each of the photographed objects;

[0008] Generating a first facial image of each of the photographed objects according to the facial image and facial key points of each of the photographed objects;

[0009] A target image is generated according to the target video frame in the video clip and the first face image of each of the photographed objects.

[0010] Another aspect of the present application provides an image processing device, including:

[0011] An acquisition module, used to acquire a video clip shot by a camera; wherein the video clip displays at least one face of a subject;

[0012] A detection module, used to perform face detection and key point detection on each video frame in the video clip to obtain a face image and face key points corresponding to each of the photographed objects;

[0013] A first generating module, used for generating a first face image of each of the photographed objects according to the face image and face key points of each of the photographed objects;

[0014] The second generating module is used to generate a target image according to the target video frame in the video clip and the first face image of each of the photographed objects.

[0015] Another aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method as described in the aforementioned aspect is implemented.

[0016] Another aspect of the present application provides a chip, which includes an interface circuit and a processing circuit coupled to each other, wherein the interface circuit is used to input or output a signal, and the processing circuit is configured to execute the image processing method as described in the aforementioned aspect.

[0017] Another aspect of the present application is a non-temporary computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the image processing method as described in the aforementioned aspect is implemented.

[0018] Another aspect of the present application provides a computer program product on which a computer program is stored. When the program is executed by a processor, the image processing method as described in the above aspect is implemented.

[0019] The image processing method, device, electronic device, chip and storage medium proposed in the present application, in a shooting scene, a video clip is recorded by a camera, and a high-quality face image of each shooting object is generated according to the face image and face key points of each shooting object in the video clip (recorded as the first face image in the present application, exemplarily, the eyes of the shooting object in the first face image are in a fully open state, and / or, the face of the shooting object in the first face image is facing or facing the camera), and then the high-quality first face image of each shooting object is fused with the target video frame in the video clip to obtain the target image, which can not only improve the generation quality of the target image and improve the user's shooting experience, but also do not need to wait for the time when all shooting objects open their eyes, which can improve the flexibility and generalization of the method. Exemplarily, when the number of shooting objects is multiple, the target image can be a group photo, which can improve the user's shooting experience in the group photo scene; when the number of shooting objects is one, the target image can be a single person photo, which can improve the user's shooting experience in the single person shooting scene.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A schematic diagram of a flow chart of a first image processing method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of a flow chart of a second image processing method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a flow chart of a third image processing method provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a fourth image processing method provided in an embodiment of the present application;

[0026] Figure 5 Schematic diagram of the implementation principle of each embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of the structure of a chip proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0030] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] Compared with face detection and face recognition, face synthesis is a more open and dynamic research field. This field not only covers graphics technologies such as modeling, deformation and rendering, but also deeply explores the cutting-edge technology of image generation in deep learning. Although different face synthesis methods may follow similar frameworks and technical routes, they vary greatly in specific details and implementation methods. This diversity provides a broad space for research and innovation in the field of face synthesis.

[0032] In the related art, in a group photo scenario, the following face synthesis scheme can be used to improve the quality of the group photo: a face image with eyes open is captured from multiple frames of images captured by a camera, and the captured face image is merged with the background image to generate a group photo with eyes fully open.

[0033] However, the above solution directly maps the face image onto the background image, which will cause lighting errors and image defects. In addition, if there is no face image with open eyes in the multiple frames of images captured by the camera, the eyes in the group photo finally generated by this solution will still be closed.

[0034] Alternatively, in a group photo scenario, it is possible to detect whether all users' eyes are open. If so, all users are photographed to ensure that no closed-eye problems are found in the photos. However, this method has at least the following problems: if someone keeps closing their eyes during the group photo, the photo cannot be taken for a long time, and the photo production efficiency is extremely low.

[0035] Therefore, in response to at least one of the problems existing in the above-mentioned related technologies, the present application proposes an image processing method, device, electronic device, chip and storage medium.

[0036] The image processing method, device, electronic device, chip and storage medium of the embodiments of the present application are described below with reference to the accompanying drawings.

[0037] Figure 1 A schematic flowchart of the first image processing method provided in an embodiment of the present application.

[0038] It should be noted that the image processing method of the embodiment of the present application can be applied to an image processing device. In some possible embodiments, the image processing device can be configured in an electronic device or a chip so that the electronic device or the chip can perform an image processing function. In addition, in some possible embodiments, the image processing device can also be software in an electronic device.

[0039] In any embodiment of the present application, the chip can be integrated into an electronic device. Among them, the chip includes a central processing unit (CPU), an image signal processing (ISP), an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field programmable gate array (FPGA), a system on a chip (SOC), a reduced instruction set computer (RISC), etc., which are not listed one by one this time.

[0040] like Figure 1 As shown, the image processing method may include the following steps S101 to S104:

[0041] Step S101, obtaining a video clip shot by a camera; wherein the video clip displays at least one face of a subject.

[0042] There is no restriction on the type of camera. Exemplarily, the camera includes a front camera and a rear camera.

[0043] The shooting object includes the shooting subject, wherein the shooting subject includes the most important and most attention-grabbing core person in the shooting picture. Exemplarily, the shooting object may include various user objects within the shooting angle of the camera.

[0044] There is no limit on the length of the video clip, nor on the number of video frames contained in the video clip. For example, in order to reduce the waiting time of the user and improve the efficiency of image output, the length of the video clip should not be set too long. For example, the number of video frames contained in the video clip is greater than or equal to 2.

[0045] In the embodiment of the present application, each object can be photographed or recorded by a camera to obtain a video clip.

[0046] As an application scenario, when there are multiple subjects to be photographed, that is, in a group photo scenario, multiple frames of images can be continuously captured by a camera to obtain a video clip showing multiple subjects.

[0047] Step S102: performing face detection and key point detection on each video frame in the video clip to obtain a face image and face key points corresponding to each photographed object.

[0048] In an embodiment of the present application, face detection and face key point detection can be performed on each video frame in the video clip based on a face detection algorithm and a face key point detection algorithm to obtain a face image and face key points of each subject in each video frame.

[0049] As an example, the facial images of each subject in each video frame can be arranged according to the shooting time of each video frame to obtain a facial image sequence of each subject; and the facial key points of each subject in each video frame can be arranged according to the shooting time of each video frame to obtain a facial key point sequence of each subject.

[0050] The facial image sequence includes a group of ordered facial images, wherein the arrangement order of each facial image is determined according to the shooting time of the video frame where the facial image is located.

[0051] The facial key point sequence includes a set of ordered key point data, wherein the key point data includes the identifier (such as index) and two-dimensional coordinates of the facial key points; wherein the two-dimensional coordinates refer to the coordinate positions of the facial key points in the video frame.

[0052] Exemplarily, the number of video frames included in the marked video clip is N frames, then the facial image sequence of each subject may include N facial images of the subject; since a face contains 68 facial key points (17 key points for depicting the facial contour, 14 key points for depicting eyebrows, 12 key points for depicting eyes, 9 key points for depicting the nose, and 16 key points for depicting the mouth), the facial key point sequence of each subject may include key point data of N*68 facial key points of the subject.

[0053] Step S103: generating a first facial image of each photographed object according to the facial image and facial key points of each photographed object.

[0054] Among them, the first facial image can be a high-quality facial image. For example, the eyes of the subject in each first facial image are fully open, and / or the face of the subject in the first facial image is facing or looking directly at the camera.

[0055] In an embodiment of the present application, for any one of the at least one photographed subjects, deep learning technology can be used to generate a new facial image of the subject based on the facial image and facial key points of the subject in each video frame, which is recorded as the first facial image in the present application.

[0056] Exemplarily, the facial image sequence and facial key point sequence of the photographed subject may be input into an image generation model for image processing to obtain a first facial image of the photographed subject output by the image generation model.

[0057] There is no restriction on the model structure of the image generation model. By way of example, the image generation model includes but is not limited to: Recursive Neural Network (RNN for short), Convolutional Neural Network (CNN for short), Transformer, etc.

[0058] Step S104: generating a target image according to the target video frame in the video clip and the first face image of each photographed object.

[0059] The target video frame may be any video frame in the video clip. For example, in order to improve the image quality, that is, to improve the quality of the final image generation, the highest quality video frame in the video clip may be used as the target video frame.

[0060] In an embodiment of the present application, a target image may be generated based on an image processing technology or an image synthesis technology according to a target video frame in a video clip and first facial images of multiple subjects.

[0061] The image processing method of the embodiment of the present application, in a shooting scene, records a video clip through a camera, and generates a high-quality face image of each shooting object (recorded as the first face image in the present application, exemplarily, the eyes of the shooting object in the first face image are in a fully open state, and / or, the face of the shooting object in the first face image is facing or looking directly at the camera), and then fuses the high-quality first face image of each shooting object with the target video frame in the video clip to obtain the target image, which can not only improve the generation quality of the target image and improve the user's shooting experience, but also does not need to wait for the time when all the shooting objects open their eyes, which can improve the flexibility and generalization of the method. Exemplarily, when the number of shooting objects is multiple, the target image can be a group photo, which can improve the user's shooting experience in the group photo scene; when the number of shooting objects is one, the target image can be a single person photo, which can improve the user's shooting experience in the single person shooting scene.

[0062] The present application embodiment provides another image processing method. Figure 2 A schematic flowchart of the second image processing method provided in an embodiment of the present application.

[0063] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0064] like Figure 2 As shown, the image processing method may include the following steps S201 to S206:

[0065] Step S201, obtaining a video clip shot by a camera; wherein the video clip displays at least one face of a subject.

[0066] Step S202: performing face detection and key point detection on each video frame in the video clip to obtain a face image and face key points corresponding to each photographed object.

[0067] Step S203: generating a first facial image of each photographed object according to the facial image and facial key points of each photographed object.

[0068] It should be noted that the explanation of steps S201 to S203 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0069] Step S204: determining a target video frame from the video frames according to the quality score of each video frame in the video segment.

[0070] The quality score is used to indicate the shooting quality of the video frame.

[0071] In an embodiment of the present application, the quality score of each video frame in the video clip can be calculated. For example, a No-Reference Image Quality Assessment (NR-IQA) algorithm can be used to calculate the quality score of each video frame. After that, a target video frame can be determined from each video frame based on the quality score of each video frame. For example, the video frame with the highest quality score can be used as the target video frame.

[0072] In any embodiment of the present application, the following steps A to C may be used to calculate the quality score of each video frame:

[0073] Step A: for the i-th (i is a positive integer not greater than N) frame of video, extract the multi-scale features of the i-th frame of video; wherein the multi-scale features are used to characterize the basic attributes and statistical characteristics of the i-th frame of video, such as contrast, edge direction distribution, etc.

[0074] Step B: Calculate the degree of deviation between the multi-scale features of the i-th video frame and the statistical model; wherein the statistical model is used to indicate the distribution of the multi-scale features of multiple sample images (ie, normal natural images), that is, to characterize the statistical features of normal natural images.

[0075] As an example, the Kullback-Leibler (KL) divergence between the probability density function of the multi-scale features of the i-th video frame and the statistical model can be calculated, and the KL divergence can be used as the degree of deviation between the multi-scale features of the i-th video frame and the statistical model.

[0076] Step C: Based on the degree of deviation, determine the quality score of the i-th video frame.

[0077] Among them, the smaller the deviation degree is, the closer the multi-scale features of the i-th frame of video are to the multi-scale features of the natural image, and the better the image quality of the i-th frame of video; conversely, the larger the deviation degree is, the worse the image quality of the i-th frame of video.

[0078] That is, the quality score of the i-th video frame is negatively correlated with the above-mentioned deviation degree, that is, the smaller the deviation degree, the higher the quality score, and conversely, the greater the deviation degree, the lower the quality score.

[0079] In any embodiment of the present application, the following steps A' to C' may be used to calculate the quality score of each video frame:

[0080] Step A': for the i-th (i is a positive integer not greater than N) video frame, extract the local contrast feature and edge definition feature of the i-th video frame.

[0081] As an example, local contrast features may be extracted by calculating local variances of multiple image blocks in the i-th video frame, and edge clarity features may be extracted by calculating the contrast between edge pixels and background pixels in the i-th video frame.

[0082] Step B': Fuse and normalize the local contrast features and edge definition features to obtain the target features.

[0083] As an example, the local contrast feature and the edge clarity feature may be fused to obtain a fused feature, and the fused feature may be normalized to remove the mean and standardize the variance to obtain the target feature.

[0084] Step C': Perform quality prediction on the target feature to obtain the quality score of the i-th video frame.

[0085] As an example, the target feature may be input into a pre-trained regression model for quality prediction, and the regression model outputs a predicted quality score to reflect the quality level of the i-th video frame.

[0086] In any embodiment of the present application, the following steps A′ to C′ may be used to calculate the quality score of each video frame:

[0087] Step A": for the i-th (i is a positive integer not greater than N) frame of video, calculate the index value of the i-th frame of video under multiple image quality indexes.

[0088] The image quality indicators include, but are not limited to: Entropy (used to calculate the entropy of an image, a higher entropy value indicates that the image contains more information), Laplacian Variance (used to calculate the Laplace variation of an image, a larger Laplace variance value generally indicates that the image is clearer), etc.

[0089] Step B": Obtain the image sharpness of the i-th video frame.

[0090] As an example, the Laplacian variance of the i-th video frame may be used to evaluate the image sharpness of the i-th video frame.

[0091] Step C": Calculate the quality score of the i-th video frame according to the index values ​​and image sharpness under multiple image quality indexes.

[0092] As an example, a weighted sum of index values ​​and image sharpness under a plurality of image quality indexes may be performed to obtain a quality score of the i-th video frame.

[0093] In summary, different methods can be used to calculate the quality score of each video frame, which can improve the flexibility and applicability of the method.

[0094] It should be noted that the above-mentioned method of calculating the quality score of the video frame is only an example, but the present application is not limited to this. In actual application, other algorithms can also be used to calculate the quality score of each video frame. For example, feature extraction technology can also be used to extract image features of the video frame, and a trained deep learning model can be used to score the image features to obtain the quality score of the video frame. The embodiments of the present application are not limited to this.

[0095] Step S205 , determining the facial posture of each shooting object in the target video frame according to the facial key points of each shooting object in the target video frame.

[0096] In an embodiment of the present application, for any one of the at least one photographed subjects, a face detection algorithm can be used to perform face detection on the target video frame to obtain a face image of the subject, and a face key point detection algorithm can be used to perform key point detection on the face image to obtain all the face key points of the subject, so that the face posture of the subject can be calculated based on all the face key points of the subject.

[0097] As a possible implementation, the following steps D to F may be used to calculate the face posture of each photographed object:

[0098] Step D: For any one of the at least one photographed object, obtain the two-dimensional coordinates of all facial key points of the photographed object in the target video frame.

[0099] Exemplarily, the two-dimensional coordinates may be the two-dimensional coordinates of the facial key points (ie, the two-dimensional key points) in the image coordinate system or the pixel coordinate system of the target video frame.

[0100] Step E: Determine key points from the three-dimensional face model that match the facial key points of the photographed subject.

[0101] In which, each two-dimensional facial key point of the photographed object has a corresponding matching key point (i.e., a three-dimensional key point) in the predefined three-dimensional face model. Exemplarily, the indexes between the two-dimensional facial key points and the three-dimensional key points are one-to-one corresponding, i.e., the i-th two-dimensional facial key point corresponds to the i-th three-dimensional key point.

[0102] Step F: Based on the two-dimensional coordinates of the facial key points of the photographed subject and the three-dimensional coordinates of the matching key points, determine the facial posture of the photographed subject in the target video frame.

[0103] Exemplarily, the camera posture parameters (such as rotation matrix and translation matrix) can be solved based on the two-dimensional coordinates of each two-dimensional facial key point of the photographed object and the three-dimensional coordinates of the corresponding matching three-dimensional key points through algorithms such as the perspective n-point problem (Perspective-n-Point, abbreviated as PnP), where the camera posture parameters are used to describe the posture of the photographed object relative to the camera. In the present application, the camera posture parameters can be converted into the facial posture of the photographed object.

[0104] As another possible implementation, the following steps D' to F' may be used to calculate the face posture of each photographed object:

[0105] Step D': for any one of the at least one photographed object, obtain a heat map associated with the facial key points of the photographed object; wherein the pixel value of each pixel in the heat map is used to indicate the confidence that the corresponding pixel is a facial key point.

[0106] Among them, each two-dimensional facial key point corresponds to a heat map. For example, when a face contains 68 facial key points, 68 heat maps can be generated, wherein the pixel value of each pixel point in each heat map is used to indicate the confidence that the corresponding pixel point is the corresponding two-dimensional facial key point.

[0107] Exemplarily, a deep learning model can be used to generate a heat map corresponding to each two-dimensional facial key point.

[0108] Step E': Determine the target pixel in the heat map according to the pixel value of each pixel in the heat map.

[0109] Exemplarily, for a heat map corresponding to any two-dimensional facial key point, the local maximum value or peak value in the heat map can be used as the target pixel point.

[0110] Step F': Determine the facial posture of the subject in the target video frame according to the two-dimensional coordinates of the target pixel in the heat map and the three-dimensional coordinates of the key points in the three-dimensional face model that match the facial key points of the subject.

[0111] In which, each two-dimensional facial key point of the photographed object has a corresponding matching key point (i.e., a three-dimensional key point) in the predefined three-dimensional face model. Exemplarily, the indexes between the two-dimensional facial key points and the three-dimensional key points are one-to-one corresponding, i.e., the i-th two-dimensional facial key point corresponds to the i-th three-dimensional key point.

[0112] Exemplarily, the camera posture parameters (such as rotation matrix and translation matrix) can be solved through algorithms such as PnP based on the three-dimensional coordinates of the three-dimensional key points in the three-dimensional face model that match each two-dimensional facial key point of the photographed object, and the two-dimensional coordinates of the target pixel points in the heat map corresponding to each two-dimensional facial key point. The camera posture parameters are used to describe the posture of the photographed object relative to the camera. In this application, the camera posture parameters can be converted into the facial posture of the photographed object.

[0113] It should be noted that the above-mentioned method of calculating facial posture is only an exemplary description, but the present application is not limited to this. In actual application, other algorithms can also be used to calculate the facial posture of each photographed object. For example, a regression-based method (such as support vector machine, random forest, deep neural network and other algorithms) can also be used to predict facial posture according to two-dimensional facial key points. The embodiment of the present application does not limit this.

[0114] Step S206, generating a target image according to the target video frame, the first face image of each photographed object, and the face posture.

[0115] In an embodiment of the present application, a target image can be generated based on image processing according to the target video frame, the first face image and face posture of each photographed object.

[0116] As a possible implementation method, a target image is generated, for example, as follows: for any one of at least one photographed subject, first, based on the first facial image and facial posture of the subject, a second facial image matching the facial posture can be generated. Exemplarily, based on the facial posture of the subject, the first facial image of the subject can be adjusted or rendered to obtain a second facial image matching the facial posture of the subject.

[0117] Afterwards, the second face image of each photographed object can be fused with the target video frame to obtain the target image. Exemplarily, deep learning technology can be used to fuse the second face image of each photographed object with the target video frame to obtain the target image. For example, the second face image of each photographed object and the target video frame can be input into a UNet (a convolutional neural network) model for image fusion to obtain the target image output by the UNet model.

[0118] Therefore, using neural networks for face generation can avoid lighting errors and image defects caused by direct mapping and background fusion, thereby improving the quality of image generation.

[0119] The image processing method of the embodiment of the present application generates a target image by integrating the facial posture of each subject in the target video frame with the best quality, the high-quality first facial image of each subject, and the target video frame itself. This can ensure that the facial posture of each subject in the target image is highly consistent with the facial posture in the original target video frame on the basis of improving the generation quality of the target image, that is, ensure that the generated target image is natural and realistic, and maintains the realism of the original shooting scene, which not only improves the overall quality of the image, but also retains more original details to meet the personalized shooting needs of different users.

[0120] The present application embodiment provides another image processing method. Figure 3 A schematic flowchart of the third image processing method provided in an embodiment of the present application.

[0121] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0122] like Figure 3 As shown, the image processing method may include the following steps S301 to S307:

[0123] Step S301, obtaining a video clip shot by a camera; wherein the video clip displays at least one face of a subject.

[0124] It should be noted that the explanation of step S301 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0125] Step S302: performing face detection on each video frame in the video clip to obtain the face position of each subject in each video frame.

[0126] The face position of each photographed subject refers to the two-dimensional position of the face of the photographed subject in the video frame.

[0127] In an embodiment of the present application, face detection may be performed on each video frame in a video clip based on a face detection algorithm to obtain the face position of each subject in each video frame.

[0128] Step S303 , based on the face position of each subject in each video frame, an initial face image of each subject is captured from each video frame.

[0129] In an embodiment of the present application, for any frame of video, a facial image of each subject can be captured from the video frame based on the face position of each subject in the video frame, which is referred to as an initial facial image in the present application.

[0130] Step S304 , arranging the initial face images of the same subject in each video frame according to the shooting time of each video frame to obtain a face image sequence of the same subject.

[0131] In an embodiment of the present application, the initial facial images of the same subject in each video frame may be arranged according to the shooting time of each video frame in the video clip to obtain a facial image sequence of the same subject.

[0132] Exemplarily, the initial facial images of the same subject in each video frame may be sorted from small to large according to the shooting time to obtain a facial image sequence of the same subject.

[0133] Step S305 , performing key point detection on each image in the facial image sequence of the same photographed subject to obtain a facial key point sequence of the same photographed subject.

[0134] In an embodiment of the present application, key point detection can be performed on each image in a facial image sequence of the same subject based on a facial key point detection algorithm to obtain the facial key points in each image, and based on the facial key points in each image, a facial key point sequence of the same subject can be generated.

[0135] Step S306: generating a first face image of each shooting object according to the face image sequence and the face key point sequence of each shooting object.

[0136] Exemplarily, for any one of the at least one photographed subjects, a facial image sequence and a facial key point sequence of the photographed subject may be input into an image generation model for image processing to obtain a first facial image of the photographed subject output by the image generation model.

[0137] There is no restriction on the model structure of the image generation model. By way of example, the image generation model includes but is not limited to: RNN, CNN, Transformer, etc.

[0138] Step S307: Generate a target image according to the target video frame in the video clip and the first face image of each photographed object.

[0139] It should be noted that the explanation of steps S306 to S307 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0140] The image processing method of the embodiment of the present application performs face clipping based on the face position of each photographed subject, and generates a face image sequence and a face key point sequence for each photographed subject based on the clipped face image, which can improve the effectiveness and accuracy of sequence acquisition.

[0141] The present application embodiment provides another image processing method. Figure 4 A schematic flowchart of the fourth image processing method provided in an embodiment of the present application.

[0142] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0143] like Figure 4 As shown, the image processing method may include the following steps S401 to S408:

[0144] Step S401, obtaining a video clip shot by a camera; wherein the video clip displays at least one face of a subject.

[0145] Step S402: performing face detection on each video frame to obtain the face position of each subject in each video frame.

[0146] Step S403 , based on the face position of each subject in each video frame, an initial face image of each subject is captured from each video frame.

[0147] Step S404 , arranging the initial face images of the same subject in each video frame according to the shooting time of each video frame to obtain a face image sequence of the same subject.

[0148] It should be noted that the explanation of steps S401 to S404 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0149] Step S405 , based on the input requirements of the image generation model, resize each image in the facial image sequence of the same photographed subject to obtain a target image sequence of the same photographed subject.

[0150] The input requirements include, but are not limited to, requirements on the size, format, and color space of the input image.

[0151] Among them, the image generation model is used to generate facial images. It should be noted that the present application does not limit the model structure of the image generation model. Exemplarily, the image generation model includes but is not limited to: RNN, CNN, Transformer, etc.

[0152] In an embodiment of the present application, input requirements that are compatible with the image generation model can be obtained, and based on the input requirements, the image size of each image in a facial image sequence of the same photographed object can be adjusted so that the adjusted image size of each image meets the input requirements, thereby obtaining a target image sequence of the same photographed object.

[0153] Step S406: perform key point detection on each image in the target image sequence of the same photographed subject to obtain a facial key point sequence of the same photographed subject.

[0154] In an embodiment of the present application, key point detection can be performed on each image in a target image sequence of the same subject based on a facial key point detection algorithm to obtain facial key points in each image, and based on the facial key points in each image, a facial key point sequence of the same subject can be generated.

[0155] Step S407, inputting the target image sequence and the facial key point sequence of each photographed object into the image generation model for image processing, and obtaining the first facial image of each photographed object output by the image generation model.

[0156] In any embodiment of the present application, the eyes of the subject in the first facial image are fully open, and / or the face of the subject in the first facial image is facing the camera.

[0157] In an embodiment of the present application, the target image sequence and the facial key point sequence corresponding to the same photographed object can be simultaneously input into the image generation model for processing to obtain the first facial image of the same photographed object output by the image generation model.

[0158] Step S408: generating a target image according to the target video frame in the video clip and the first face image of each photographed object.

[0159] It should be noted that the explanation of step S408 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0160] The image processing method of the embodiment of the present application uses a neural network to generate the required first face image, which has higher flexibility and output image quality.

[0161] In any embodiment of the present application, the number of photographed subjects is taken as multiple for illustrative explanation. In a group photo scenario, when the image captured by the camera contains multiple faces, the present application can record a video clip, analyze the faces in the video clip, use a neural network for modeling, generate the best quality open-eye face image under the current scene and lighting, and then merge the open-eye face image with the background image to obtain a high-quality group photo in which all photographed subjects have their eyes open.

[0162] For example, the implementation principle of the technical solution provided by this application can be as follows: Figure 5 As shown, it mainly includes the following steps:

[0163] 1. Perform face detection on the captured image to obtain the number of faces. If the number of faces in the captured image is less than 2, it is considered to be a non-group photo scene. At this time, the algorithm can be exited. If the number of faces in the captured image is greater than or equal to 2, a video clip is shot in a group photo scene.

[0164] 2. Use a face detection algorithm (including but not limited to Multi-task Cascaded Convolutional Networks (MTCNN), You Only Look Once or Yolo, etc.) to perform face detection on each frame in the video clip to obtain the face position of each subject, and based on the face position of each subject, take a screenshot of the video frame to obtain a face image of each subject.

[0165] 3. Perform key point detection on the face image of the same subject in each video frame to construct a face image sequence and a face key point sequence of the same subject.

[0166] Exemplarily, each face image in the face image sequence can be resized, and a face key point detection algorithm (including but not limited to: Digital Library (DLIB, which provides a variety of machine learning and numerical calculation functions, including powerful face key point detection tools), etc.) can be used to detect 68 face key points in each face image, so that a face key point sequence can be generated based on the 68 face key points in each face image.

[0167] 4. Input the facial image sequence and facial key point sequence of each photographed subject into the image generation model (such as RNN) at the same time, and use the image generation model to generate a frontal and eye-open facial image of each photographed subject (referred to as the first facial image in this application).

[0168] 5. Use a no-reference image quality assessment algorithm to calculate the quality score of each video frame in the video clip, and select the target video frame with the highest quality score.

[0169] 6. Estimating key points of the face of the same subject in the target video frame, and calculating the face posture of the same subject based on the face key points.

[0170] 7. Use the frontal and eye-opening facial image of the same subject generated in step 4 to render an image of the facial posture of the same subject to obtain a rendered facial image (referred to as the second facial image in this application).

[0171] 8. Input the rendered face image and the original content of the target video frame into UNet for content fusion to obtain the group photo output by Unet (referred to as the target image in this application).

[0172] In summary, there is no need to wait for all subjects to open their eyes, which can improve the flexibility and generalization of the method. In addition, using a neural network to generate the required open-eye face images has higher flexibility and output image quality.

[0173] In order to implement the above embodiment, the embodiment of the present application also proposes an image processing device.

[0174] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0175] like Figure 6 As shown, the image processing device 600 may include: an acquisition module 610 , a detection module 620 , a first generation module 630 and a second generation module 640 .

[0176] The acquisition module 610 is used to acquire a video clip captured by a camera; wherein the video clip displays at least one face of a subject;

[0177] A detection module 620 is used to perform face detection and key point detection on each video frame in the video clip to obtain a face image and face key points corresponding to each photographed object;

[0178] A first generating module 630, configured to generate a first facial image of each photographed object according to the facial image and facial key points of each photographed object;

[0179] The second generating module 640 is used to generate a target image according to the target video frame in the video clip and the first face image of each photographed object.

[0180] Furthermore, in an implementation method of an embodiment of the present application, the second generation module 640 is used to: determine a target video frame from each video frame based on the quality score of each video frame in the video clip; determine the facial posture of each subject in the target video frame based on the facial key points of each subject in the target video frame; generate a target image based on the target video frame, the first facial image and facial posture of each subject.

[0181] In one implementation of the embodiment of the present application, the second generation module 640 is used to: generate a second facial image of each subject based on the first facial image and facial posture of each subject; and fuse the second facial image of each subject with the target video frame to obtain a target image.

[0182] In one implementation of the embodiment of the present application, the second generation module 640 is used to render a first facial image of any photographed subject according to the facial posture of any photographed subject to obtain a second facial image matching the facial posture.

[0183] In one implementation of an embodiment of the present application, the second generation module 640 is used to: obtain the two-dimensional coordinates of the facial key points of any photographed object in the target video frame; determine the key points that match the facial key points of any photographed object from the three-dimensional face model; and determine the facial posture of any photographed object in the target video frame based on the two-dimensional coordinates of the facial key points of any photographed object and the three-dimensional coordinates of the matching key points.

[0184] In one implementation of the embodiment of the present application, the second generation module 640 is used to: obtain a heat map associated with the facial key points of any photographed object; wherein the pixel value of each pixel in the heat map is used to indicate the confidence that the corresponding pixel is a facial key point; determine the target pixel from the heat map based on the pixel value of each pixel in the heat map; determine the facial posture of any photographed object in the target video frame based on the two-dimensional coordinates of the target pixel in the heat map and the three-dimensional coordinates of the key points in the three-dimensional face model that match the facial key points of any photographed object.

[0185] In one implementation of the embodiment of the present application, the second generation module 640 is also used to perform any of the following: extracting multi-scale features of any video frame, and calculating the degree of deviation between the multi-scale features of any video frame and the statistical model, so as to determine the quality score of any video frame based on the degree of deviation; wherein the statistical model is used to indicate the distribution of multi-scale features of multiple sample images; extracting local contrast features and edge clarity features of any video frame, and fusing and normalizing the local contrast features and edge clarity features to obtain target features, and performing quality prediction on the target features to obtain the quality score of any video frame; calculating the index values ​​of any video frame under multiple image quality indicators, and obtaining the image sharpness of any video frame, so as to calculate the quality score of any video frame according to the index values ​​and image sharpness under multiple image quality indicators.

[0186] In one implementation of the embodiment of the present application, the detection module 620 is used to: perform face detection on each video frame to obtain the face position of each subject in each video frame; based on the face position of each subject in each video frame, capture the initial face image of each subject from each video frame; according to the shooting time of each video frame, arrange the initial face images of the same subject in each video frame to obtain a face image sequence of the same subject; perform key point detection on each image in the face image sequence of the same subject to obtain a face key point sequence of the same subject.

[0187] In one implementation of the embodiment of the present application, the detection module 620 is used to: resize each image in the facial image sequence of the same photographed object based on the input requirements of the image generation model to obtain a target image sequence of the same photographed object; perform key point detection on each image in the target image sequence of the same photographed object to obtain a facial key point sequence of the same photographed object;

[0188] The first generation module 630 is used to: input the target image sequence and the facial key point sequence of each photographed object into the image generation model for image processing, and obtain the first facial image of each photographed object output by the image generation model.

[0189] In one implementation of the embodiment of the present application, the eyes of the subject in the first facial image are fully open, and / or the face of the subject in the first facial image is facing the camera.

[0190] It should be noted that the above explanation of the embodiment of the image processing method is also applicable to the image processing device of this embodiment, and will not be repeated here.

[0191] In the image processing device of the embodiment of the present application, in a shooting scene, a video clip is recorded by a camera, and a high-quality face image of each shooting object is generated according to the face image and face key points of each shooting object in the video clip (recorded as the first face image in the present application, exemplarily, the eyes of the shooting object in the first face image are in a fully open state, and / or, the face of the shooting object in the first face image is facing or looking directly at the camera), and then the high-quality first face image of each shooting object is fused with the target video frame in the video clip to obtain the target image, which can not only improve the generation quality of the target image and improve the user's shooting experience, but also do not need to wait for the time when all shooting objects open their eyes, which can improve the flexibility and generalization of the method. Exemplarily, when the number of shooting objects is multiple, the target image can be a group photo, which can improve the user's shooting experience in the group photo scene; when the number of shooting objects is one, the target image can be a single person photo, which can improve the user's shooting experience in the single person shooting scene.

[0192] In order to implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method as described in any of the above embodiments is implemented.

[0193] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. For example, the electronic device 700 may be a vehicle, a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0194] Reference Figure 7 The electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0195] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0196] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0197] The power component 706 provides power to the various components of the electronic device 700. The power component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 700.

[0198] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0199] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0200] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0201] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of the components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or a component of the electronic device 700, the presence or absence of contact between the user and the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0202] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0203] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above method.

[0204] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of an electronic device 700 to complete the above method. For example, the non-transitory computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0205] In order to implement the above embodiments, the present application also proposes a chip, wherein the chip includes an interface circuit and a processing circuit coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is configured to execute the image processing method provided in any of the above embodiments.

[0206] Figure 8 Schematic diagram of a chip structure proposed in the embodiment of the present application. Figure 8 The structure of the chip 800 is shown, but is not limited to this.

[0207] The chip 800 includes a processing circuit 801 , and the processing circuit 801 is configured to execute any of the above image processing methods.

[0208] In some embodiments, the chip 800 further includes one or more interface circuits 802. Optionally, the interface circuit 802 is connected to the memory 803, and the interface circuit 802 can be used to receive signals from the memory 803 or other devices, and the interface circuit 802 can be used to send signals to the memory 803 or other devices. For example, the interface circuit 802 can read instructions stored in the memory 803 and send the instructions to the processing circuit 801.

[0209] In some embodiments, the interface circuit 802 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 801 performs other steps.

[0210] In some embodiments, terms such as interface circuit, interface, transceiver pin, and transceiver may be used interchangeably.

[0211] In some embodiments, the chip 800 further includes one or more memories 803 for storing instructions. Optionally, all or part of the memory 803 may be outside the chip 800.

[0212] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method as described in any of the above method embodiments is implemented.

[0213] In order to implement the above embodiments, the present application also proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in any of the above method embodiments is implemented.

[0214] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0215] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0216] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0217] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic devices), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0218] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA for short), a field programmable gate array (FPGA for short), etc.

[0219] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0220] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0221] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that: include: Obtain a video clip captured by a camera; wherein the video clip displays at least one face of a subject; Performing face detection and key point detection on each video frame in the video clip to obtain a face image and face key points corresponding to each of the photographed objects; Generating a first facial image of each of the photographed objects according to the facial image and facial key points of each of the photographed objects; A target image is generated according to the target video frame in the video clip and the first face image of each of the photographed objects.

2. The method according to claim 1, characterized in that The step of generating a target image according to the target video frame in the video clip and the first face image of each of the photographed objects includes: Determining the target video frame from each of the video frames according to the quality score of each of the video frames in the video clip; Determining the facial posture of each of the photographed subjects in the target video frame according to the facial key points of each of the photographed subjects in the target video frame; The target image is generated according to the target video frame, the first face image and the face posture of each of the photographed objects.

3. The method according to claim 2, characterized in that The step of generating the target image according to the target video frame, the first face image of each of the photographed objects, and the face posture includes: Generating a second facial image of each photographed object according to the first facial image and facial posture of each photographed object; The second face image of each of the photographed objects is fused with the target video frame to obtain the target image.

4. The method according to claim 3, characterized in that Generating a second face image of each of the photographed objects according to the first face image and face posture of each of the photographed objects comprises: According to the facial posture of any photographed object, a first facial image of any photographed object is rendered to obtain a second facial image matching the facial posture.

5. The method according to claim 2, characterized in that: The step of determining the facial posture of each of the photographed subjects in the target video frame according to the facial key points of each of the photographed subjects in the target video frame comprises: Obtain the two-dimensional coordinates of the facial key points of any photographed object in the target video frame; Determining key points from the three-dimensional face model that match key points of the face of any of the photographed subjects; Based on the two-dimensional coordinates of the facial key points of any photographed object and the three-dimensional coordinates of the matched key points, the facial posture of any photographed object in the target video frame is determined.

6. The method according to claim 2, characterized in that The step of determining the facial posture of each of the photographed subjects in the target video frame according to the facial key points of each of the photographed subjects in the target video frame comprises: Obtain a heat map associated with a key point of a face of any photographed object; wherein the pixel value of each pixel in the heat map is used to indicate the confidence that the corresponding pixel is the key point of the face; Determine a target pixel in the heat map according to the pixel value of each pixel in the heat map; The facial posture of any photographed object in the target video frame is determined according to the two-dimensional coordinates of the target pixel point in the heat map and the three-dimensional coordinates of the key points in the three-dimensional face model that match the facial key points of any photographed object.

7. The method according to claim 2, characterized in that The quality score of each of the video frames is determined by any of the following: Extracting multi-scale features of any video frame, and calculating the degree of deviation between the multi-scale features of any video frame and a statistical model, so as to determine a quality score of any video frame based on the degree of deviation; wherein the statistical model is used to indicate the distribution of multi-scale features of a plurality of sample images; Extracting local contrast features and edge clarity features of any video frame, fusing and normalizing the local contrast features and the edge clarity features to obtain target features, and performing quality prediction on the target features to obtain a quality score of any video frame; Calculate the index values ​​of any video frame under multiple image quality indicators, and obtain the image sharpness of any video frame, so as to calculate the quality score of any video frame according to the index values ​​under the multiple image quality indicators and the image sharpness.

8. The method according to any one of claims 1 to 7, characterized in that The performing face detection and key point detection on each video frame in the video clip to obtain a face image and a face key point corresponding to each of the photographed objects includes: Performing face detection on each of the video frames to obtain the face position of each of the photographed subjects in each of the video frames; Based on the face position of each of the photographed subjects in each of the video frames, intercepting an initial face image of each of the photographed subjects from each of the video frames; Arranging the initial face images of the same shooting object in each of the video frames according to the shooting time of each of the video frames to obtain a face image sequence of the same shooting object; Key point detection is performed on each image in the facial image sequence of the same photographed subject to obtain a facial key point sequence of the same photographed subject.

9. The method according to claim 8, characterized in that The step of performing key point detection on each image in the facial image sequence of the same photographed object to obtain a facial key point sequence of the same photographed object includes: Based on the input requirements of the image generation model, resizing each image in the face image sequence of the same photographed object to obtain a target image sequence of the same photographed object; Performing key point detection on each image in the target image sequence of the same photographed object to obtain a facial key point sequence of the same photographed object; The step of generating a first face image of each of the photographed objects according to the face image and face key points of each of the photographed objects comprises: The target image sequence and the facial key point sequence of each of the photographed objects are input into the image generation model for image processing, so as to obtain the first facial image of each of the photographed objects output by the image generation model.

10. The method according to any one of claims 1 to 7, characterized in that The eyes of the subject in the first facial image are fully open, and / or the face of the subject in the first facial image is facing the camera.

11. An image processing device, characterized in that: include: An acquisition module, used to acquire a video clip shot by a camera; wherein the video clip displays at least one face of a subject; A detection module, used to perform face detection and key point detection on each video frame in the video clip to obtain a face image and face key points corresponding to each of the photographed objects; A first generating module, used for generating a first face image of each of the photographed objects according to the face image and face key points of each of the photographed objects; The second generating module is used to generate a target image according to the target video frame in the video clip and the first face image of each of the photographed objects.

12. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A chip, characterized in that: The chip comprises an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output a signal, and the processing circuit is used to implement the method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images