Image frame selection method and apparatus, electronic device, and storage medium

Abstract

The application provides an image frame selection method and device, electronic equipment and storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a plurality of image frames; performing global feature analysis on the plurality of image frames to obtain a global score of each image frame; performing local feature analysis on a face region and a human body region in each first image frame to obtain a local score of each image frame; performing fusion processing on the global score and the local score of each image frame to obtain a comprehensive score of each image frame; and determining a target image frame from the plurality of image frames according to the comprehensive score of each image frame. The comprehensive score is obtained based on the global score and the local score, so that the comprehensive score of each image frame is more accurate. In addition, the comprehensive score is corrected, and the target image frame is determined based on the corrected comprehensive score, so that the image frames can be de-duplicated, the target image frame with the best effect can be determined, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image frame selection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With a large number of video users, capturing memorable moments from videos is a great way to preserve those beautiful moments. Automatically finding user-favorite images from a large pool of pictures, reducing the workload of manual frame selection and minimizing storage space usage, has become a hot demand in current photography apps.

[0003] In related technologies, a quality assessment network is used to predict the sharpness of each frame in the captured video, and then the sharpest frame or multiple frames are selected as the automatically selected image frames.

[0004] However, in related technologies, it is difficult to select the best image frame based on image sharpness, which reduces the user experience. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing an image frame selection method, apparatus, electronic device, and storage medium to solve the aforementioned technical problems in the related art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0007] In a first aspect, embodiments of the present invention provide an image frame selection method, the method comprising:

[0008] Acquire multiple image frames, wherein at least one first image frame of the multiple image frames includes: a face region and a body region;

[0009] Global feature analysis is performed on the multiple image frames to obtain a global score for each image frame.

[0010] Local feature analysis is performed on the facial region and the human body region in each first image frame to obtain a local score for each image frame;

[0011] The global and local scores of each image frame are fused together to obtain a comprehensive score for each image frame.

[0012] The target image frame is determined from the plurality of image frames based on the overall score of each image frame.

[0013] Optionally, the step of performing local feature analysis on the facial region and the human body region in each first image frame to obtain a local score for each image frame includes:

[0014] Calculate the facial feature score for each image frame based on the facial region in each first image frame;

[0015] Based on the human body region in each first image frame, a pose score for each image frame is calculated; wherein, the local score for each image frame includes: facial feature score and pose score for each image frame.

[0016] Optionally, calculating the facial feature score for each image frame based on the facial region in each first image frame includes:

[0017] Based on the facial region in each first image frame, calculate the state, expression state, and facial appearance evaluation parameters of the preset feature parts in each first image frame;

[0018] Based on the state of preset feature parts, expression state, and facial appearance evaluation parameters in each first image frame, calculate the facial feature score for each first image frame;

[0019] Based on the facial feature scores of each first image frame, calculate the facial feature scores of the second image frame among the plurality of image frames;

[0020] Wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the facial feature score of each image frame includes the facial feature scores of each first image frame and the second image frame.

[0021] Optionally, the step of calculating the state of preset feature parts, expression state, and facial appearance evaluation parameters in each first image frame based on the facial region in each first image frame includes:

[0022] The size of the preset feature parts is calculated based on the facial key points in the facial region to obtain the state of the preset feature parts in each first image frame;

[0023] A preset facial expression recognition model is used to identify the facial region of the human face to obtain the facial expression state of each first image frame;

[0024] A preset facial appearance evaluation model is used to identify the facial region of the person, and facial appearance evaluation parameters are obtained for each first image frame.

[0025] Optionally, calculating the pose score for each image frame based on the human body region in each first image frame includes:

[0026] Key point detection is performed on the human body region in each first image frame to generate a key point heatmap for each first image frame;

[0027] A preset pose scoring model is used to detect the key point heatmap of each first image frame to obtain the pose score of each first image frame.

[0028] Based on the pose score of each first image frame, calculate the pose score of the second image frame among the plurality of image frames;

[0029] Wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the pose score of each image frame includes the pose scores of each first image frame and the second image frame.

[0030] Optionally, the step of performing global feature analysis based on the plurality of image frames to obtain a global score for each image frame in the plurality of image frames includes:

[0031] A preset aesthetic scoring model is used to perform global feature analysis on the multiple image frames to obtain the aesthetic evaluation parameters of the multiple image frames;

[0032] A preset quality scoring model is used to perform global feature analysis on the multiple image frames to obtain the quality evaluation parameters of the multiple image frames.

[0033] The global score for each of the multiple image frames includes: aesthetic evaluation parameters and quality evaluation parameters for the multiple image frames.

[0034] Optionally, determining the target image frame from the plurality of image frames based on the comprehensive score of each image frame includes:

[0035] The overall score is corrected to obtain the corrected overall score;

[0036] Based on the corrected comprehensive score, the multiple image frames are sorted to obtain a first sorting result;

[0037] The target image frame is determined from the plurality of image frames based on the first sorting result.

[0038] Optionally, the step of correcting the overall score to obtain the corrected overall score includes:

[0039] Based on the comprehensive score of each image frame, the multiple image frames are sorted to obtain a second sorting result;

[0040] Calculate the similarity between any two image frames in the plurality of image frames;

[0041] Based on the second ranking result and the similarity, the comprehensive score is corrected to obtain the corrected comprehensive score.

[0042] In a second aspect, embodiments of the present invention provide an image frame selection device, the device comprising:

[0043] The acquisition module is used to acquire multiple image frames, wherein at least one first image frame of the multiple image frames includes: a face region and a human body region;

[0044] The analysis module is used to perform global feature analysis based on the plurality of image frames to obtain a global score for each image frame; and to perform local feature analysis based on the facial region and the human body region in each first image frame to obtain a local score for each image frame.

[0045] The processing module is used to perform fusion processing based on the global score and local score of each image frame to obtain the comprehensive score of each image frame;

[0046] The determination module is used to determine the target image frame from the plurality of image frames based on the comprehensive score of each image frame.

[0047] Optionally, the analysis module is specifically used to calculate the facial feature score of each image frame based on the facial region in each first image frame; and to calculate the pose score of each image frame based on the human body region in each first image frame; wherein the local score of each image frame includes: the facial feature score and the pose score of each image frame.

[0048] Optionally, the analysis module is specifically used to calculate the state, expression state, and facial appearance evaluation parameters of preset feature parts in each first image frame based on the facial region in each first image frame; calculate the facial feature score of each first image frame based on the state, expression state, and facial appearance evaluation parameters of preset feature parts in each first image frame; and calculate the facial feature score of a second image frame among the plurality of image frames based on the facial feature score of each first image frame; wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the facial feature score of each image frame includes the facial feature scores of each first image frame and the second image frame.

[0049] Optionally, the analysis module is specifically used to calculate the size of preset feature parts based on the facial key points in the facial region, and obtain the state of the preset feature parts in each first image frame; to identify the facial region using a preset expression recognition model, and obtain the expression state of each first image frame; and to identify the facial region using a preset beauty evaluation model, and obtain the beauty evaluation parameters of each first image frame.

[0050] Optionally, the analysis module is specifically used to perform keypoint detection on the human body region in each first image frame to generate a keypoint heatmap for each first image frame; to use a preset pose scoring model to detect the keypoint heatmap of each first image frame to obtain a pose score for each first image frame; and to calculate the pose score of a second image frame among the plurality of image frames based on the pose scores of each first image frame; wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the pose score of each image frame includes the pose scores of each first image frame and the second image frame.

[0051] Optionally, the analysis module is specifically used to perform global feature analysis on the plurality of image frames using a preset aesthetic scoring model to obtain aesthetic evaluation parameters of the plurality of image frames; and to perform global feature analysis on the plurality of image frames using a preset quality scoring model to obtain quality evaluation parameters of the plurality of image frames; the global score of each image frame in the plurality of image frames includes: aesthetic evaluation parameters and quality evaluation parameters of the plurality of image frames.

[0052] Optionally, the determining module is specifically used to correct the comprehensive score to obtain a corrected comprehensive score; sort the plurality of image frames according to the corrected comprehensive score to obtain a first sorting result; and determine the target image frame from the plurality of image frames according to the first sorting result.

[0053] Optionally, the determining module is specifically configured to sort the plurality of image frames according to the comprehensive score of each image frame to obtain a second sorting result; calculate the similarity between every two image frames in the plurality of image frames; and correct the comprehensive score according to the second sorting result and the similarity to obtain the corrected comprehensive score.

[0054] Thirdly, embodiments of the present invention provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the image frame selection method described in any of the first aspects above.

[0055] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when the computer program is read and executed, it implements the image frame selection method described in any of the first aspects above.

[0056] The beneficial effects of this invention are as follows: This invention provides an image frame selection method, comprising: acquiring multiple image frames, wherein at least one first image frame includes a facial region and a human body region; performing global feature analysis on the multiple image frames to obtain a global score for each image frame; performing local feature analysis on the facial region and human body region in each first image frame to obtain a local score for each image frame; performing fusion processing on the global and local scores of each image frame to obtain a comprehensive score for each image frame; and determining a target image frame from the multiple image frames based on the comprehensive score of each image frame. By obtaining a comprehensive score for each image frame based on multiple aspects such as the global and local scores, the comprehensive score of each image frame is more accurate and better represents the presentation effect of the image frame. Therefore, based on the comprehensive score of each image frame, it is easier to accurately determine the target image frame with the best effect from multiple image frames, thus improving the user experience. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 1 ;

[0059] Figure 2 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 2 ;

[0060] Figure 3 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 3 ;

[0061] Figure 4 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 4 ;

[0062] Figure 5 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 5 ;

[0063] Figure 6 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 6 ;

[0064] Figure 7 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 7 ;

[0065] Figure 8 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 8 ;

[0066] Figure 9 This is a schematic diagram of the structure of an image frame selection device provided in an embodiment of the present invention;

[0067] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0069] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0070] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0071] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0073] In related technologies, a quality assessment network is used to predict the sharpness of each frame in a captured video, and then the sharpest one or more frames are selected as automatically chosen image frames. However, in these technologies, it is difficult to select the best image frames based on image sharpness, which reduces the user experience.

[0074] To address the aforementioned technical problems in related technologies, this application provides an image frame selection method. This method performs global feature analysis and local feature analysis, generating a global score and a local score for each image frame. Based on these scores, a comprehensive score for each image frame is obtained, making the comprehensive score more accurate and better representing the image frame's presentation effect. Furthermore, by using the comprehensive score of each image frame, the best-performing target image frame can be accurately determined from multiple image frames, thus improving the user experience.

[0075] This application provides an image frame selection method that is applied to an electronic device, which can be a terminal device. The terminal device can be any of the following: desktop computer, laptop computer, tablet computer, smartphone, etc. This application does not impose any specific limitations on this.

[0076] The following is an explanation of an image frame selection method provided by an embodiment of this application.

[0077] Figure 1 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 1 ,like Figure 1 As shown, the method may include:

[0078] S101. Acquire multiple image frames.

[0079] Among them, at least one first image frame of the plurality of image frames includes: a human face region and a human body region.

[0080] It should be noted that at least one first image frame is a portion of multiple image frames, and the multiple image frames may also include a second image frame that does not contain a face region or a body region; of course, at least one first image frame may also be all of the multiple image frames, that is, each of the multiple image frames has a face region and a body region, and the embodiments of this application do not impose specific limitations on this.

[0081] In this embodiment, a target video can be acquired, which includes multiple image frames in a sequential order. The target video can be a video captured by a user using a terminal device, a video pre-stored within the terminal device, or a video acquired through other methods; this embodiment does not impose specific limitations on these methods.

[0082] S102. Perform global feature analysis based on multiple image frames to obtain the global score for each image frame in the multiple image frames.

[0083] In some implementations, global feature analysis of at least one first dimension is performed on multiple image frames to obtain a global score of at least one first dimension for each image frame in the multiple image frames.

[0084] For example, at least one first dimension includes at least one of the following: aesthetic dimension, quality dimension, etc. Of course, this is just an example, and the first dimension can also be other dimensions that can be used for global feature analysis.

[0085] It is worth noting that a preset model can be used to perform global feature analysis based on multiple image frames, or a preset algorithm or preset rules can be used to perform global feature analysis based on multiple image frames, or other methods can be used to perform global feature analysis based on multiple image frames. This application does not impose specific limitations on these methods.

[0086] S103. Perform local feature analysis on the facial and human body regions in each first image frame to obtain a local score for each image frame.

[0087] The local score for each image frame includes: a local score for the facial region and a local score for the body region.

[0088] Alternatively, local feature analysis can be performed first on the facial region and then on the human body region; or local feature analysis can be performed first on the human body region and then on the facial region; or local feature analysis can be performed simultaneously on both the facial region and the human body region to obtain a local score for each image frame.

[0089] In this embodiment of the application, the face region refers to the region where the face is located in the first image frame, and the human body region refers to the region where the human body is located in the first image frame.

[0090] It should be noted that the process of S103 can be executed using a preset model, a preset algorithm or a preset rule, or other methods. This application embodiment does not impose specific restrictions on this.

[0091] It is worth noting that the process of S102 can be executed first and then the process of S103 can be executed, or the process of S103 can be executed first and then the process of S102 can be executed, or the processes of S102 and S103 can be executed simultaneously. This application embodiment does not impose specific restrictions on this.

[0092] S104. Perform fusion processing based on the global score and local score of each image frame to obtain the comprehensive score of each image frame.

[0093] In some implementations, a comprehensive score for each image frame is obtained by linearly fusing the local score for the face region, the local score for the body region, and the global score for at least one first dimension.

[0094] S105. Based on the comprehensive score of each image frame, determine the target image frame from multiple image frames.

[0095] The number of target image frames can be at least one.

[0096] In addition, based on the overall score of each image frame, a preset number of image frames with high overall scores are selected as target image frames from multiple image frames, or image frames with overall scores greater than or equal to a preset score are selected as target image frames.

[0097] In practical applications, the target image frame can be displayed to the user. This target image frame is the best-looking image frame among multiple image frames, which realizes the selection of the best-looking image frame from multiple image frames, making the automatic selection of image frames more accurate and improving the user experience.

[0098] In summary, this invention provides an image frame selection method, comprising: acquiring multiple image frames, wherein at least one first image frame includes a facial region and a human body region; performing global feature analysis on the multiple image frames to obtain a global score for each image frame; performing local feature analysis on the facial region and human body region in each first image frame to obtain a local score for each image frame; performing fusion processing on the global and local scores of each image frame to obtain a comprehensive score for each image frame; and determining a target image frame from the multiple image frames based on the comprehensive score of each image frame. By obtaining a comprehensive score for each image frame based on multiple aspects such as the global and local scores, the comprehensive score of each image frame is more accurate and better represents the presentation effect of the image frame. Therefore, based on the comprehensive score of each image frame, it is easier to accurately determine the target image frame with the best effect from multiple image frames, thus improving the user experience.

[0099] Optional, Figure 2 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 2 ,like Figure 2 As shown, the process in S103 above, which involves performing local feature analysis on the facial and human body regions in each first image frame to obtain a local score for each image frame, may include:

[0100] S201. Calculate the facial feature score for each image frame based on the facial region in each first image frame.

[0101] In some implementations, at least one type of facial feature is calculated for each image frame based on the facial region in each first image frame, and a facial feature score is calculated for each image frame based on the at least one type of facial feature.

[0102] S202. Calculate the pose score for each image frame based on the human body region in each first image frame.

[0103] The local score for each image frame includes: facial feature score and pose score for each image frame.

[0104] In this embodiment of the application, a preset posture scoring model or a preset posture scoring algorithm can be used to analyze the human posture and movement of the human body region in each first image frame and calculate the posture score of each image frame.

[0105] In addition, posture scoring is used to characterize the aesthetics of human posture and movement in human body areas, and is an important factor in the aesthetic evaluation of portrait images.

[0106] Optional, Figure 3A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 3 ,like Figure 3 As shown, the process of calculating the facial feature score of each image frame based on the facial region in each first image frame in step S201 may include:

[0107] S301. Based on the facial region in each first image frame, calculate the state, expression state, and appearance evaluation parameters of the preset feature parts in each first image frame.

[0108] In this process, face detection is performed on each of the multiple image frames. If a face is found, the image frame is designated as the first image frame; otherwise, it is designated as the second image frame.

[0109] In some implementations, the state of the preset feature part is the eye state. Facial features are analyzed in the facial region of each first image frame to determine the eye state as open, half-open, closed, blinking, or blinking one eye (wink). The expression state is determined as unnatural, normal, or happy. The appearance evaluation parameter is determined to be one of the parameters within the preset parameter range.

[0110] S302. Calculate the facial feature score for each first image frame based on the state of the preset feature parts, the expression state, and the appearance evaluation parameters in each first image frame.

[0111] In this embodiment, the weight of a preset feature part is determined based on its state, the weight of an expression is determined based on its facial expression state, and the facial appearance score is determined based on facial appearance evaluation parameters. A preset formula is used to calculate the facial feature score for each first image frame based on the weight of the preset feature part, the weight of the expression, and the facial appearance score. The preset feature part can be the eyes, or it can be the mouth; this embodiment does not impose any specific limitation on this.

[0112] The above preset formula can be: Facial feature score = Appearance score * Eye weight * Expression weight.

[0113]

[0114]

[0115] The appearance score can be a value between 1 and 100, or between 1 and 10, or a value within a preset appearance score range. This application embodiment does not impose specific limitations on this.

[0116] S303. Based on the facial feature scores of each first image frame, calculate the facial feature scores of the second image frame among multiple image frames.

[0117] The second image frame is an image frame other than the first image frame among multiple image frames, and the facial feature score of each image frame includes the facial feature scores of each first image frame and the second image frame.

[0118] It is worth noting that the average of the facial feature scores of multiple first image frames can be used as the facial feature score of the second image frame among multiple image frames. The second image frame is an image frame that does not include the facial area of ​​the human face.

[0119] Figure 4 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 4 ,like Figure 4 As shown, the process of calculating the state, expression state, and facial appearance evaluation parameters of preset feature parts in each first image frame based on the facial region in each first image frame can include:

[0120] S401. Calculate the size of the preset feature parts based on the facial key points in the facial region, and obtain the state of the preset feature parts in each first image frame.

[0121] The preset feature can be the eyes.

[0122] In some implementations, the size of the eyes is calculated based on facial key points in the facial region to obtain the eye state in each first image frame. The eye state can be any of the following: open eye state, half-open eye state, closed eye state, blinking eye state, or blinking one eye state.

[0123] S402. Use a preset expression recognition model to recognize the facial region of the face and obtain the expression state of each first image frame.

[0124] The facial expression state is defined by any one of the following: unnatural state, normal state, or happy state. The preset facial expression recognition model is a pre-trained model.

[0125] S403. Use a preset facial appearance evaluation model to identify the facial region of the face and obtain the facial appearance evaluation parameters for each first image frame.

[0126] It should be noted that the appearance evaluation parameter can be an appearance score, which is a value within a preset appearance score range. The preset appearance score range can be set according to actual needs.

[0127] Optional, Figure 5 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 5 ,like Figure 5 As shown, the process of calculating the pose score of each image frame based on the human body region in each first image frame in S202 above may include:

[0128] S501. Perform key point detection on the human body region in each first image frame to generate a key point heatmap for each first image frame.

[0129] In this process, human detection is performed on each of the multiple image frames. If a human body is present, the image frame is designated as the first image frame; otherwise, it is designated as the second image frame.

[0130] In addition, HRNet (High-Resolution Net) can be used to detect key points in the human body region of each first image frame. The key point heatmap of each first image frame can be the human skeleton key point heatmap of each first image frame. Compared with the first image frame, the human skeleton key points can reduce the interference of clothing, makeup, image background, etc.

[0131] S502. Using a preset pose scoring model, the key point heatmap of each first image frame is detected to obtain the pose score of each first image frame.

[0132] In this process, the keypoint heatmap of each first image frame is input into a preset pose scoring model, which outputs a pose score for each first image frame. The preset pose scoring model is a pre-trained model.

[0133] The training process for the pre-defined posture scoring model is as follows:

[0134] A collection of images of poses rated as good and bad by designers is used as training data. To reduce the interference of factors such as image background, clothing, and makeup on network training, this embodiment uses the sample human skeleton key point heatmap as input. An image is randomly selected from the set of good poses, denoted as image A, with a corresponding annotation score of La. An image is randomly selected from the set of bad poses, denoted as image B, with a corresponding annotation score of Lb.

[0135] Keypoint detection is performed on images A and B to obtain sample human skeleton keypoint heatmap A and sample human skeleton keypoint heatmap B. Sample human skeleton keypoint heatmap A and sample human skeleton keypoint heatmap B are then input into PoseNet (a real-time pose detection technology) to obtain evaluation scores Sa and Sb. Finally, the preset loss function is iteratively optimized to train PoseNet. The preset loss function L is shown below:

[0136]

[0137] M can be set according to actual needs; for example, M can be set to 2.

[0138] In practical applications, HRNet can be used to perform key point detection on images A and B, and MobileNetV2 (a lightweight deep neural network) with 0.5 times the number of channels can be used as the backbone network of PoseNet. Correspondingly, the preset pose scoring model can be the trained MobileNetV2.

[0139] S503. Based on the pose score of each first image frame, calculate the pose score of the second image frame among multiple image frames.

[0140] The second image frame is an image frame other than the first image frame among multiple image frames, and the pose score of each image frame includes the pose scores of each first image frame and the second image frame.

[0141] In some implementations, the average pose score of a plurality of first image frames is calculated, and this average is used as the pose score of a second image frame, which is an image frame that does not include the human body region.

[0142] In the embodiments of this application, the local score of each image frame in the plurality of image frames includes: facial feature score and pose score of each image frame.

[0143] Optional, Figure 6 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 6 ,like Figure 6 As shown, global feature analysis is performed on multiple image frames to obtain a global score for each image frame, including:

[0144] S601. Using a preset aesthetic scoring model, perform global feature analysis on multiple image frames to obtain aesthetic evaluation parameters for multiple image frames.

[0145] Among them, the aesthetic evaluation parameters can also be called aesthetic scores. Aesthetic scores mainly evaluate image frames from multiple dimensions such as image brightness, contrast, hue, and scene composition.

[0146] In some implementations, the preset aesthetic scoring model can also be called ANet. The preset aesthetic scoring model is a pre-trained model, and the training process of the preset aesthetic scoring model is as follows:

[0147] Designers rate the images in a "good" or "bad" dataset, resulting in a binary classification dataset. This dataset includes images with good aesthetic ratings and images with poor aesthetic ratings. Image A is randomly selected from the good-rated dataset, with a corresponding labeled score of La; image B is randomly selected from the poor-rated dataset, with a corresponding labeled score of Lb. Images A and B are then input into an ANet to obtain evaluation scores Sa and Sb, respectively. Finally, the ANet is trained iteratively by optimizing a pre-defined loss function, which can be:

[0148]

[0149] Where M can be 2, and MobileNetV2 with 0.5 times the number of channels can be used as the backbone network of ANet.

[0150] S602. Using a preset quality scoring model, perform global feature analysis on multiple image frames to obtain quality evaluation parameters for multiple image frames.

[0151] Among them, the quality evaluation parameter can also be called the quality score. The quality score is mainly used to evaluate whether the image has noise, blur, artifacts, etc.

[0152] In some implementations, the preset quality scoring model can also be called QNet. The preset quality scoring model is a pre-trained model, and the training process of the preset quality scoring model is as follows:

[0153] First, a batch of clear images is collected as positive samples, which are high-quality samples. Then, the positive samples are subjected to degradation processing such as Gaussian blurring, adding noise, compression, and adjusting contrast to obtain negative samples, which are low-quality samples. The training process of QNet is similar to that of ANet.

[0154] First, randomly select one image from the set of good-looking images, denoted as image A, with a corresponding labeled score of La; randomly select one image from the set of poor-looking images, denoted as image B, with a corresponding labeled score of Lb. Then, input images A and B into QNet respectively to obtain evaluation scores Sa and Sb. Finally, iteratively optimize the formula and preset loss function to train QNet.

[0155]

[0156] The loss function M is set to 2, and MobileNetV2 with 0.5 times the number of channels is used as the backbone network of QNet.

[0157] It is worth noting that the global score for each image frame in the multiple image frames includes: aesthetic evaluation parameters and quality evaluation parameters for the multiple image frames.

[0158] Optional, Figure 7 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 7 ,like Figure 7 As shown, the process of determining the target image frame from multiple image frames based on the comprehensive score of each image frame in S105 above may include:

[0159] S701. Correct the overall score to obtain the corrected overall score.

[0160] S702. Based on the corrected comprehensive score, sort the multiple image frames to obtain the first sorting result.

[0161] In some implementations, multiple image frames are sorted from high to low based on the corrected comprehensive score to obtain a first sorting result.

[0162] S703. Based on the first sorting result, determine the target image frame from multiple image frames.

[0163] In this embodiment of the application, a preset number of image frames ranked first in the first sorting result are used as target image frames; or, based on the corrected comprehensive score of each image frame, the image frames whose corrected comprehensive score is greater than or equal to a preset score among multiple image frames are used as target image frames. This embodiment of the application does not impose specific limitations on this.

[0164] Optional, Figure 8 A flowchart illustrating an image frame selection method provided in an embodiment of the present invention. Figure 8 ,like Figure 8 As shown, the process of correcting the overall score in S701 above to obtain the corrected overall score may include:

[0165] S801. Based on the comprehensive score of each image frame, sort the multiple image frames to obtain a second sorting result.

[0166] The second sorting result includes multiple image frames F={f1, f2, ..., fn} arranged from highest to lowest comprehensive score, with corresponding comprehensive scores S={s1, s2, ..., sn}.

[0167] S802. Calculate the similarity between any two image frames in multiple image frames.

[0168] In one possible implementation, MobileNetv2 with 0.5 times the number of channels, pre-trained on the ImageNet dataset (a computer vision system recognition project), is used as a feature extractor to extract the features of the last convolutional layer. Then, global max pooling is performed on each channel of the features to obtain the feature encoding of each image frame. Finally, the cosine distance between the encodings of the image frame to be detected and other image frames (other image frames are those ranked before the image frame to be detected in the second ranking result) is calculated as the similarity between the image frame to be detected and other image frames. Similarly, the similarity between any two image frames can be obtained.

[0169] S803. Based on the second ranking result and similarity, the comprehensive score is corrected to obtain the corrected comprehensive score.

[0170] In some implementations, for the image frame to be detected fi (fi∈F), the image frame to be detected is any one of multiple image frames, and the similarity {d} between it and image frames f1, f2, ... fi-1 is calculated respectively. 1 i, d 2 i,... d i-1 i}, take {d 1 i, d 2 i,... d i-1 i The maximum value in} is used as the similarity coefficient di=max(d 1 i， d 2 i，... d i-1 i Based on the similarity coefficient of fi, the comprehensive score si of fi is corrected to obtain si', where the formula for the comprehensive score correction can be: .

[0171] Where k is -2 and b is 2.84, the values ​​of k and b can be set according to actual needs, and this application embodiment does not impose specific restrictions on them.

[0172] In this embodiment of the application, the image frame with the highest similarity to the image frame to be detected can be called the similar image frame to the image frame to be detected. If the similar image frame ranks higher than the image frame to be detected in the second ranking result, the comprehensive score of the image frame to be detected is corrected by the above comprehensive score correction formula, the comprehensive score of the image frame to be detected is reduced, and the ranking of the image frame to be detected is shifted to the back. Based on the corrected comprehensive score, the multiple image frames are sorted again to obtain the first ranking result.

[0173] It should be noted that by calculating the similarity between every two image frames in multiple image frames, and correcting the comprehensive score based on the second ranking result and the similarity, the obtained comprehensive score can achieve the purpose of deduplication and avoid duplicate image frames in the target image frames identified in multiple image frames.

[0174] In summary, a comprehensive score for each image frame is obtained based on multiple aspects, including global and local scores. This comprehensive score is more accurate and better represents the presentation effect of each image frame. Furthermore, based on the comprehensive score, it is easier to accurately determine the best-performing target image frame from multiple image frames, thus improving the user experience. The global score includes aesthetic and quality scores, while the local score includes facial feature and pose scores. Evaluating image frames comprehensively based on multiple scores results in a more accurate comprehensive score. First, a pre-trained network is used to extract feature codes from the images. Then, the cosine distance between the feature codes of different images is calculated to obtain the similarity between two images. Finally, the comprehensive score is corrected based on the similarity to achieve deduplication.

[0175] Moreover, the image frame selection method provided in this embodiment of the invention can perform real-time frame selection for client users, selecting the most exciting and attractive frames from videos captured in real time or imported later, reducing the workload of manual selection and storage space occupation for users, and giving users a sense of surprise.

[0176] The following describes the image frame selection apparatus, electronic device, and storage medium used to implement the image frame selection method provided in this application. For the specific implementation process and technical effects, please refer to the relevant content of the above-mentioned image frame selection method, which will not be repeated below.

[0177] Figure 9 This is a schematic diagram of the structure of an image frame selection device provided in an embodiment of the present invention, as shown below. Figure 9 As shown, the device includes:

[0178] The acquisition module 901 is used to acquire multiple image frames, wherein at least one first image frame of the multiple image frames includes: a face region and a human body region;

[0179] The analysis module 902 is used to perform global feature analysis based on the plurality of image frames to obtain a global score for each image frame in the plurality of image frames; and to perform local feature analysis based on the facial region and the human body region in each first image frame to obtain a local score for each image frame.

[0180] The processing module 903 is used to perform fusion processing based on the global score and local score of each image frame to obtain the comprehensive score of each image frame;

[0181] The determination module 904 is used to determine the target image frame from the plurality of image frames based on the comprehensive score of each image frame.

[0182] Optionally, the analysis module 902 is specifically used to calculate the facial feature score of each image frame based on the facial region in each first image frame; and to calculate the pose score of each image frame based on the human body region in each first image frame; wherein the local score of each image frame includes: the facial feature score and the pose score of each image frame.

[0183] Optionally, the analysis module 902 is specifically used to calculate the state, expression state, and facial appearance evaluation parameters of preset feature parts in each first image frame based on the facial region in each first image frame; calculate the facial feature score of each first image frame based on the state, expression state, and facial appearance evaluation parameters of preset feature parts in each first image frame; and calculate the facial feature score of a second image frame among the plurality of image frames based on the facial feature score of each first image frame; wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the facial feature score of each image frame includes the facial feature scores of each first image frame and the second image frame.

[0184] Optionally, the analysis module 902 is specifically used to calculate the size of a preset feature part based on the facial key points in the facial region, and obtain the state of the preset feature part in each first image frame; to identify the facial region using a preset expression recognition model, and obtain the expression state of each first image frame; and to identify the facial region using a preset beauty evaluation model, and obtain the beauty evaluation parameters of each first image frame.

[0185] Optionally, the analysis module 902 is specifically used to perform key point detection on the human body region in each first image frame to generate a key point heatmap for each first image frame; to use a preset pose scoring model to detect the key point heatmap of each first image frame to obtain a pose score for each first image frame; and to calculate the pose score of a second image frame among the plurality of image frames based on the pose scores of each first image frame; wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the pose score of each image frame includes the pose scores of each first image frame and the second image frame.

[0186] Optionally, the analysis module 902 is specifically used to perform global feature analysis on the plurality of image frames using a preset aesthetic scoring model to obtain aesthetic evaluation parameters of the plurality of image frames; and to perform global feature analysis on the plurality of image frames using a preset quality scoring model to obtain quality evaluation parameters of the plurality of image frames; the global score of each image frame in the plurality of image frames includes: aesthetic evaluation parameters and quality evaluation parameters of the plurality of image frames.

[0187] Optionally, the determining module 904 is specifically used to correct the comprehensive score to obtain a corrected comprehensive score; sort the plurality of image frames according to the corrected comprehensive score to obtain a first sorting result; and determine the target image frame from the plurality of image frames according to the first sorting result.

[0188] Optionally, the determining module 904 is specifically used to sort the plurality of image frames according to the comprehensive score of each image frame to obtain a second sorting result; calculate the similarity between every two image frames in the plurality of image frames; and correct the comprehensive score according to the second sorting result and the similarity to obtain the corrected comprehensive score.

[0189] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0190] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0191] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 10 As shown, the electronic device includes: a processor 1001 and a memory 1002.

[0192] The memory 1002 is used to store programs, and the processor 1001 calls the programs stored in the memory 1002 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described again here.

[0193] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, is used to perform the above-described method embodiments.

[0194] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0197] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image frame selection method, characterized in that, The method includes: Acquire multiple image frames, wherein at least one first image frame of the multiple image frames includes: a human face region and a human body region; Global feature analysis is performed on the multiple image frames to obtain a global score for each image frame. Local feature analysis is performed on the facial region and the human body region in each first image frame to obtain a local score for each image frame; The global and local scores of each image frame are fused together to obtain a comprehensive score for each image frame. The target image frame is determined from the plurality of image frames based on the comprehensive score of each image frame; The step of performing local feature analysis on the facial region and the human body region in each first image frame to obtain a local score for each image frame includes: Based on the facial region in each first image frame, calculate the state, expression state, and facial appearance evaluation parameters of the preset feature parts in each first image frame; Based on the state of preset feature parts, expression state, and facial appearance evaluation parameters in each first image frame, calculate the facial feature score for each first image frame; Based on the facial feature scores of each first image frame, calculate the facial feature scores of the second image frame among the plurality of image frames; Wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the facial feature score of each image frame includes the facial feature scores of each first image frame and the second image frame; Based on the human body region in each first image frame, a pose score for each image frame is calculated; wherein, the local score for each image frame includes: facial feature score and pose score for each image frame; The step of performing global feature analysis based on the multiple image frames to obtain a global score for each image frame includes: A preset aesthetic scoring model is used to perform global feature analysis on the multiple image frames to obtain the aesthetic evaluation parameters of the multiple image frames; A preset quality scoring model is used to perform global feature analysis on the multiple image frames to obtain the quality evaluation parameters of the multiple image frames. The global score for each of the plurality of image frames includes: aesthetic evaluation parameters and quality evaluation parameters for the plurality of image frames; The step of determining the target image frame from the plurality of image frames based on the comprehensive score of each image frame includes: The overall score is corrected to obtain the corrected overall score; Based on the corrected comprehensive score, the multiple image frames are sorted to obtain a first sorting result; The target image frame is determined from the plurality of image frames based on the first sorting result.

2. The method according to claim 1, characterized in that, The step of calculating the state, expression state, and facial appearance evaluation parameters of preset feature parts in each first image frame based on the facial region in each first image frame includes: The size of the preset feature parts is calculated based on the facial key points in the facial region of the face, and the state of the preset feature parts in each first image frame is obtained. A preset facial expression recognition model is used to identify the facial region of the human face to obtain the facial expression state of each first image frame; A preset facial appearance evaluation model is used to identify the facial region of the person, and facial appearance evaluation parameters are obtained for each first image frame.

3. The method according to claim 1, characterized in that, The step of calculating the pose score for each image frame based on the human body region in each first image frame includes: Key point detection is performed on the human body region in each first image frame to generate a key point heatmap for each first image frame; A preset pose scoring model is used to detect the key point heatmap of each first image frame to obtain the pose score of each first image frame. Based on the pose score of each first image frame, calculate the pose score of the second image frame among the plurality of image frames; Wherein, the second image frame is an image frame other than the first image frame among the plurality of image frames, and the pose score of each image frame includes the pose scores of each first image frame and the second image frame.

4. The method according to claim 1, characterized in that, The process of correcting the overall score to obtain the corrected overall score includes: Based on the comprehensive score of each image frame, the multiple image frames are sorted to obtain a second sorting result; Calculate the similarity between any two image frames in the plurality of image frames; Based on the second ranking result and the similarity, the comprehensive score is corrected to obtain the corrected comprehensive score.

5. An image frame selection device, characterized in that, The device includes: The acquisition module is used to acquire multiple image frames, wherein at least one first image frame of the multiple image frames includes: a face region and a human body region; The analysis module is used to perform global feature analysis based on the plurality of image frames to obtain a global score for each image frame; and to perform local feature analysis based on the facial region and the human body region in each first image frame to obtain a local score for each image frame. The processing module is used to perform fusion processing based on the global score and local score of each image frame to obtain the comprehensive score of each image frame; The determination module is used to determine the target image frame from the plurality of image frames based on the comprehensive score of each image frame; The analysis module is specifically used to calculate the state, expression state, and attractiveness evaluation parameters of preset feature parts in each first image frame based on the facial region in each first image frame; calculate the facial feature score of each first image frame based on the state, expression state, and attractiveness evaluation parameters of preset feature parts in each first image frame; calculate the facial feature score of a second image frame in a plurality of image frames based on the facial feature score of each first image frame; wherein, the second image frame is an image frame other than the first image frame in the plurality of image frames, and the facial feature score of each image frame includes the facial feature scores of each first image frame and the second image frame; calculate the posture score of each image frame based on the human body region in each first image frame; wherein, the local score of each image frame includes: the facial feature score and the posture score of each image frame; The analysis module is specifically used to perform global feature analysis on the multiple image frames using a preset aesthetic scoring model to obtain aesthetic evaluation parameters for the multiple image frames; and to perform global feature analysis on the multiple image frames using a preset quality scoring model to obtain quality evaluation parameters for the multiple image frames; the global score for each image frame in the multiple image frames includes: aesthetic evaluation parameters and quality evaluation parameters for the multiple image frames; The determining module is specifically used to correct the comprehensive score to obtain a corrected comprehensive score; sort the plurality of image frames according to the corrected comprehensive score to obtain a first sorting result; and determine the target image frame from the plurality of image frames according to the first sorting result.

6. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the image frame selection method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when read and executed, implements the image frame selection method according to any one of claims 1-4.

Images