Method, device, medium and computing equipment for extracting key images from image sequence

Abstract

The embodiments of the present invention provide a method, apparatus, medium and computing device for extracting key images in an image sequence, the method comprising: iteratively executing the following image sampling detection process, determining the candidate images whose face detection results meet preset rules from a number of candidate frames extracted from the image sequence as key images in the image sequence: obtaining the face detection result corresponding to the current frame candidate image output by the face detection model; determining the image sampling interval corresponding to the current frame candidate image based on the face detection result; extracting the number of images separated from the current frame candidate image in the image sequence and the next frame candidate image matching the image sampling interval from the image sequence; using the next frame candidate image as a new current frame candidate image and inputting it into the face detection model so that the face detection model performs face detection on the next frame candidate image. The present disclosure can improve the extraction efficiency and accuracy of key images.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer application technology. More specifically, embodiments of the present invention relate to a method, apparatus, medium and computing device for extracting key images from an image sequence. Background Art

[0002] This section is intended to provide a background or context for embodiments of the invention. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] With the continuous development of multimedia technology, the number of image sequences containing multiple frames, such as videos, is increasing day by day. Image sequences are usually a series of images acquired in sequence and continuously at different times and orientations for a target. For example, a video can be a series of images acquired in a time sequence for a moving target over a period of time.

[0004] Image sequences can provide rich information, such as the object itself, the motion state and trajectory of the object, etc. However, image sequences also contain a lot of redundant information. In practical applications, we often encounter situations where we do not want or cannot process every frame in an image sequence. In this case, we hope to extract one or more key frames from the image sequence as key images (also called key frames) so that only these key images can be processed. Among them, key images can usually be images in the image sequence that have some outstanding features in content compared to other images, or images with eye-catching content. Summary of the invention

[0005] In this context, embodiments of the present invention are intended to provide a method, apparatus, medium and computing device for extracting key images from an image sequence.

[0006] In a first aspect of an embodiment of the present invention, a method for extracting a key image in an image sequence is provided, the method comprising:

[0007] Iteratively executing the image sampling detection process to extract a number of candidate image frames from the image sequence, and performing face detection on the number of candidate image frames;

[0008] Determine the candidate images whose face detection results meet the preset rules among the plurality of frames of candidate images as key images in the image sequence;

[0009] The image sampling detection process includes:

[0010] Obtain the face detection result output by the face detection model corresponding to the candidate image of the current frame;

[0011] Based on the face detection result, determining an image sampling interval corresponding to the current frame candidate image;

[0012] Extracting a next frame candidate image corresponding to the current frame candidate image from the image sequence; wherein the number of images between the next frame candidate image and the current frame candidate image in the image sequence matches the image sampling interval;

[0013] The next frame candidate image is used as a new current frame candidate image and input into the face detection model, so that the face detection model performs face detection on the next frame candidate image.

[0014] In a second aspect of the embodiments of the present invention, a device for extracting a key image in an image sequence is provided, the device comprising:

[0015] A detection module, used for iteratively executing an image sampling detection process to extract a number of candidate image frames from an image sequence, and performing face detection on the number of candidate image frames;

[0016] A determination module, used to determine the candidate images whose face detection results meet the preset rules among the plurality of frames of candidate images as key images in the image sequence;

[0017] Wherein, the detection module comprises:

[0018] The acquisition submodule is used to obtain the face detection result output by the face detection model corresponding to the candidate image of the current frame;

[0019] A determination submodule, configured to determine an image sampling interval corresponding to the candidate image of the current frame based on the face detection result;

[0020] An extraction submodule, configured to extract a next frame candidate image corresponding to the current frame candidate image from the image sequence; wherein the number of images between the next frame candidate image and the current frame candidate image in the image sequence matches the image sampling interval;

[0021] The detection submodule is used to input the next frame candidate image as a new current frame candidate image into the face detection model so that the face detection model performs face detection on the next frame candidate image.

[0022] In a third aspect of the embodiments of the present disclosure, a medium is provided on which a computer program is stored. When the computer program is executed by a processor, the method for extracting and processing key images in any of the above-mentioned image sequences is implemented.

[0023] In a fourth aspect of the embodiments of the present disclosure, there is provided a computing device, comprising:

[0024] processor;

[0025] a memory for storing a processor executable program;

[0026] The processor implements the above-mentioned method for extracting key images from any one of the image sequences by running the executable program.

[0027] According to an implementation manner of the present disclosure, by iteratively executing an image sampling detection process, a candidate image whose face detection result meets a preset rule among several candidate frames extracted from an image sequence can be determined as a key image in the image sequence; and when executing the image sampling detection process, specifically based on the face detection result corresponding to the current frame candidate image output by the face detection model, the image sampling interval corresponding to the current frame candidate image can be determined, and based on the image interval, the next frame candidate image corresponding to the current frame candidate image can be extracted from the image sequence and input into the face detection model as a new current frame candidate image for face detection.

[0028] In this way, the number of images between the current frame candidate image and the next frame candidate image can be dynamically adjusted, thereby avoiding the need to calculate each frame image in the image sequence or multiple frames extracted at equal intervals, reducing the amount of calculation and improving the efficiency of key image extraction.

[0029] Furthermore, since the key image is determined by determining whether the face detection rule conforms to the preset rule, the extraction accuracy of the key image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0031] Figure 1 A schematic diagram schematically shows a user interface according to an embodiment of the present disclosure;

[0032] Figure 2 A schematic diagram schematically shows an application scenario of image processing according to an embodiment of the present disclosure;

[0033] Figure 3 A flowchart of an image processing method according to an embodiment of the present disclosure is schematically shown;

[0034] Figure 4 A schematic diagram of 49 facial key points according to an embodiment of the present disclosure is schematically shown;

[0035] Figure 5 A schematic diagram of 68 facial key points according to an embodiment of the present disclosure is schematically shown;

[0036] Figure 6 A schematic diagram schematically shows an image according to an embodiment of the present disclosure;

[0037] Figure 7 A schematic diagram of a medium according to an embodiment of the present disclosure is schematically shown;

[0038] Figure 8 A block diagram schematically shows an image processing device according to an embodiment of the present disclosure;

[0039] Fig. 9 A schematic diagram of a computing device according to an embodiment of the present disclosure is schematically shown.

[0040] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0041] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0043] According to an embodiment of the present invention, a method, an apparatus, a medium and a computing device for extracting a key image in an image sequence are proposed.

[0044] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0045] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0047] In practical applications, we often encounter situations where we need to extract key images from image sequences such as videos.

[0048] Take a video APP (Application) or music APP that can provide users with movies, TV series, MVs (Music Videos) and other videos as an example. In this type of application, relevant information of multiple videos is usually output through the user interface, so that users can select videos of interest from these videos for viewing based on their relevant information.

[0049] When outputting the relevant information of each video through the user interface, a frame image related to the video is usually used as the video cover of the video, and the video cover and the video title of the video are displayed in the user interface, such as Figure 1 As shown. The video cover is usually a key image in the video. In this case, it is necessary to extract the key image from each video so as to use the extracted key image as the video cover of the video.

[0050] In the related art, for an image sequence, key images can usually be extracted from the image sequence by analyzing multiple frames of images contained in the image sequence.

[0051] Specifically, any two adjacent frames of images in the multiple frames of images included in the above image sequence can be analyzed, and the difference or similarity of the two frames of images in a specific dimension (for example, shooting target, shooting background, etc.) can be calculated, and the calculated difference or similarity can be compared with a preset threshold. If the difference or similarity is greater than the threshold, it means that the difference between the two frames of images is large; if the difference or similarity is less than the threshold, it means that the difference between the two frames of images is small. In this case, images with smaller differences can be filtered out from the multiple frames of images, and one or more frames of images with larger differences can be selected from the multiple frames of images as key images in the image sequence.

[0052] For the multiple frames of images contained in the above-mentioned image sequence, the multiple frames of images can be multiple frames of images extracted from the image sequence at equal intervals according to a preset image sampling interval. In order to ensure the adaptability of the key image extracted from the image sequence, the image sampling interval cannot usually be set too large. In this case, the two adjacent frames of images in the multiple frames of images can be the two frames of images that are closest in order in the image sequence. For example: assuming that the image sequence contains a total of 5 frames of images, namely, image 1, image 2, image 3, image 4, and image 5, which are acquired in sequence, and the image sampling interval is 2, then the three frames of images, namely, image 1, image 3 (1+2=3), and image 5 (3+2=5), can be extracted from the image sequence at equal intervals; among these three frames of images, image 1 and image 3 can be regarded as two adjacent frames of images, and image 3 and image 5 can also be regarded as two adjacent frames of images.

[0053] Alternatively, the multiple frames of images included in the above-mentioned image sequence may be all the images included in the image sequence. In this case, two adjacent frames of images in the multiple frames of images may be two adjacent frames of images in the order of the image sequence. For example, assuming that the image sequence includes three frames of images, namely, image 1, image 2, and image 3, which are acquired in sequence, then among the three frames of images, image 1 and image 2 may be regarded as two adjacent frames of images, and image 2 and image 3 may also be regarded as two adjacent frames of images.

[0054] However, in the related art, since in the process of extracting the key image from the above-mentioned image sequence, it is necessary to calculate each frame image in the image sequence or multiple frames of images extracted at equal intervals, the amount of calculation is usually large, resulting in low efficiency in extracting the key image; further, since the key image is determined by comparing the calculated results such as the difference or similarity with the threshold, and the threshold is usually set manually in advance, it is extremely important whether the manually set threshold is appropriate. Once the threshold is inappropriate, it will have a great impact on the extraction result of the key image.

[0055] In order to solve the above problems, the present disclosure provides a technical solution for extracting key images in an image sequence. In the technical solution, the image sampling detection process can be iteratively executed to determine the candidate images whose face detection results meet the preset rules among several candidate frames extracted from the image sequence as key images in the image sequence; and when executing the image sampling detection process, the image sampling interval corresponding to the current candidate frame image can be determined based on the face detection result corresponding to the current candidate frame image output by the face detection model, and based on the image interval, the next candidate frame image corresponding to the current candidate frame image is extracted from the image sequence and input into the face detection model as a new current candidate frame image for face detection.

[0056] In this way, the number of images between the current frame candidate image and the next frame candidate image can be dynamically adjusted, thereby avoiding the need to calculate each frame image in the image sequence or multiple frames extracted at equal intervals, reducing the amount of calculation and improving the efficiency of key image extraction.

[0057] Furthermore, since the key image is determined by determining whether the face detection rule conforms to the preset rule, the extraction accuracy of the key image can be improved.

[0058] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0059] Application Scenario Overview

[0060] refer to Figure 2 , Figure 2 A schematic diagram schematically shows an application scenario of image processing according to an embodiment of the present disclosure.

[0061] like Figure 2 As shown, in the application scenario of image processing, a server and at least one client (eg, client 1-N) accessing the server through any type of wired or wireless network may be included.

[0062] The above-mentioned server can be deployed on a server including an independent physical host or a server cluster composed of multiple independent physical hosts; or, the above-mentioned server can be a server built based on cloud computing services.

[0063] The above-mentioned client may correspond to an application installed by the user in the device used by the user; wherein the device may specifically be a smart phone, a tablet computer, a laptop computer, a PC (Personal Computer), a PDA (Personal Digital Assistants), a wearable device (for example: smart glasses, smart watches, etc.), a smart car device or a game console, etc.

[0064] The device equipped with the above-mentioned server or the above-mentioned client can also be equipped with shooting hardware such as an embedded camera or an external camera for collecting images or videos. In this case, these shooting hardware can be called to collect images or videos for the shooting target to obtain an image sequence corresponding to the collected multiple frames of images or videos themselves.

[0065] The server or the client may be equipped with a face detection model to perform face detection on a specific image in the image sequence based on the face detection model, and determine the key image in the image sequence based on the face detection result corresponding to the specific image.

[0066] Exemplary Methods

[0067] Combine the following Figure 2 For application scenarios, refer to Figure 3-Figure 6 The method for extracting key images in an image sequence according to an exemplary embodiment of the present disclosure is described. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0068] refer to Figure 3 , Figure 3 The flowchart of a method for extracting key images in an image sequence according to an embodiment of the present disclosure is schematically shown.

[0069] The method for extracting key images from the above-mentioned image sequence can be applied to the above-mentioned server or the above-mentioned client equipped with a machine learning model for face detection (which may be referred to as a face detection model).

[0070] For the above-mentioned server and the above-mentioned client, usually, the hardware resources of the device where the server is located are richer than the hardware resources of the device where the client is located. In this case, the computing power of the server is stronger than that of the client, and the complexity of the machine learning model that can be run in the server is also higher than the complexity of the machine learning model that can be run in the client.

[0071] In addition, for machine learning models that need to perform real-time computing tasks (for example, real-time image processing tasks during video calls, real-time audio processing tasks during voice calls, etc.), the computing rate requirements for such machine learning models are usually high, that is, such machine learning models are required to be able to output calculation results in a shorter time.

[0072] Therefore, the face detection model running in the above-mentioned server can be a full-scale machine learning model; the face detection model running in the above-mentioned client can be a lightweight machine learning model. Among them, the full-scale machine learning model can be a machine learning model obtained by training a preset machine learning model for face detection; the lightweight machine learning model can be a machine learning model obtained by trimming repeated structures such as convolutional layers in the full-scale machine learning model.

[0073] Specifically, the preset machine learning model for face detection can be trained first to obtain a full-scale machine learning model. Subsequently, through knowledge distillation, the lightweight machine learning model can learn the knowledge of the trained full-scale machine learning model, so that the model effect of the lightweight machine learning model is similar to the model effect of the full-scale machine learning model. That is, the model parameters migrated from the full-scale machine learning model through knowledge distillation can be used as the model parameters of the lightweight machine learning model.

[0074] The method for extracting key images from the above image sequence may include the following steps:

[0075] Step 301: iteratively execute the image sampling detection process to extract a number of candidate image frames from an image sequence, and perform face detection on the number of candidate image frames.

[0076] The image sampling detection process may include the following steps:

[0077] Step 3011: Obtain the face detection result output by the face detection model corresponding to the candidate image of the current frame.

[0078] Step 3012: Based on the face detection result, determine the image sampling interval corresponding to the current frame candidate image.

[0079] Step 3013: extracting a next frame candidate image corresponding to the current frame candidate image from the image sequence; wherein the number of images between the next frame candidate image and the current frame candidate image in the image sequence matches the image sampling interval.

[0080] Step 3014: The next frame candidate image is used as a new current frame candidate image and input into the face detection model so that the face detection model performs face detection on the next frame candidate image.

[0081] In practical applications, people are usually more likely to be used as shooting targets, and image sequences are collected for people. For example, movies, TV series, MVs and other videos usually use actors as shooting targets, and image sequences are collected for actors. Therefore, when performing key image extraction in an image sequence, an image with a better effect of presenting a human face can be extracted from the image sequence as the key image in the image sequence.

[0082] In this embodiment, for an image sequence from which key images need to be extracted, an image sampling detection process can be iteratively performed on the image sequence to extract a number of frames of images (referred to as candidate images) from the image sequence, and face detection can be performed on the extracted candidate images.

[0083] The following is an explanation of an image sampling detection process.

[0084] First, a frame of image can be extracted from the above image sequence as a current frame candidate image, and the current frame candidate image can be input into the above face detection model so that the face detection model performs face detection on the current frame candidate image. In this case, the face detection result corresponding to the current frame candidate image output by the face detection model can be obtained.

[0085] Secondly, based on the face detection result corresponding to the above-mentioned current frame candidate image, the image sampling interval corresponding to the current frame candidate image can be determined.

[0086] Again, in the above image sequence, a frame of image whose number of images between the above current frame candidate image and the above image sampling interval matches can be determined, and this frame of image can be extracted as the next frame candidate image corresponding to the current frame candidate image. That is, the next frame candidate image corresponding to the current frame candidate image is actually determined by the face detection result corresponding to the current frame candidate image.

[0087] Finally, the next frame candidate image can be input into the face detection model so that the face detection model performs face detection on the next frame candidate image. In this case, the next frame candidate image becomes a new current frame candidate image.

[0088] It should be noted that, for the above-mentioned next frame candidate image, in addition to requiring that the number of images between the next frame candidate image and the above-mentioned current frame candidate image in the above-mentioned image sequence matches the above-mentioned image sampling interval, it can also be required that the order of the next frame candidate image in the image sequence is located after the order of the current frame candidate image in the image sequence, or that the order of the next frame candidate image in the image sequence is located before the order of the current frame candidate image in the image sequence. Alternatively, the two frames of images before and after, whose number of images between the above-mentioned current frame candidate image and the above-mentioned image sampling interval matches, can be used as the next frame candidate images corresponding to the above-mentioned current frame candidate image; at this time, the two frames of next frame candidate images can be respectively input into the above-mentioned face detection model for face detection, that is, the two frames of next frame candidate images can be respectively used as new current frame images. The present disclosure does not impose any restrictions on this.

[0089] For example, assuming that the above image sequence contains a total of 5 frames of images, namely Image 1, Image 2, Image 3, Image 4, and Image 5, which are acquired in sequence, and the image sampling interval corresponding to Image 3 is determined to be 2 based on the face detection result corresponding to Image 3 as the current frame candidate image, then Image 5 (3+2=5) can be used as the next frame candidate image corresponding to Image 3, or Image 1 (3-2=1) can be used as the next frame candidate image corresponding to Image 3, or both Image 1 and Image 5 can be used as the next frame candidate images corresponding to Image 3.

[0090] It should also be noted that the number of images between the next frame candidate image and the current frame candidate image in the image sequence matches the image sampling interval, which may be that the difference between the order of the next frame candidate image in the image sequence and the order of the current frame candidate image in the image sequence is equal to the image sampling interval, or the number of images between the next frame candidate image and the current frame image in the image sequence is equal to the image sampling interval. The present disclosure is not limited to this.

[0091] In one embodiment shown, the image sequence may be a video. Since different videos usually have different frame rates, in order to adapt to different video frame rates, when determining the image sampling interval corresponding to the current frame candidate image based on the face detection result corresponding to the current frame candidate image, the image sampling rate corresponding to the face detection result may be determined first, and then the product of the image sampling rate and the frame rate of the video may be determined as the image sampling interval corresponding to the current frame candidate image, as shown in the following formula:

[0092] X = fps × v

[0093] Wherein, X represents the image sampling interval corresponding to the candidate image of the current frame, fps represents the frame rate of the video, and v represents the image sampling rate corresponding to the face detection result of the candidate image of the current frame. For example, assuming that the frame rate of the video is 60 frames per second, and the image sampling rate corresponding to the face detection result of the candidate image of the current frame is 0.1, it can be determined that the image sampling interval corresponding to the candidate image of the current frame is 6 (60*0.1=6) frames.

[0094] When the above image sampling detection process is iteratively performed on the above image sequence, specifically, an image with an order of 1 in the image sequence (i.e., the first frame image in the image sequence) can be first extracted from the image sequence as the first frame candidate image. Alternatively, a frame image can be first randomly extracted from the above image sequence as the first frame candidate image. The present disclosure does not limit this.

[0095] For example, assuming that the above image sequence contains a total of five frames of images, namely Image 1, Image 2, Image 3, Image 4, and Image 5, which are acquired in sequence, then Image 1 with the order of 1 in the image sequence can be extracted from the image sequence as the first frame candidate image, or Image 2 can be randomly extracted as the first frame candidate image.

[0096] Subsequently, the first frame candidate image can be input into the face detection model, so that the face detection result corresponding to the first frame candidate image output by the face detection model can be obtained, and the image sampling interval corresponding to the first frame candidate image can be determined based on the face detection result, and the second frame candidate image whose number of images spaced apart from the first frame candidate image matches the image sampling interval can be extracted from the image sequence.

[0097] When the second frame candidate image is extracted, the second frame candidate image can be input into the face detection model, so that the face detection result corresponding to the second frame candidate image output by the face detection model can be obtained, and the image sampling interval corresponding to the second frame candidate image is determined based on the face detection result, and a third frame candidate image whose number of images spaced apart from the second frame candidate image matches the image sampling interval is extracted from the image sequence.

[0098] This process is repeated until the last candidate image is extracted from the above image sequence. For the last candidate image, in the image sequence, there is no image whose number of intervals between the last candidate image and the last candidate image matches the image sampling interval corresponding to the last candidate image.

[0099] For example, assuming that the above image sequence contains a total of 5 frames of images, namely, image 1, image 2, image 3, image 4, and image 5, which are acquired in sequence, and image 1 is extracted from the image sequence as the first frame candidate image, then image 1 can be input into the above face detection model, and the face detection result corresponding to image 1 output by the face detection model can be obtained; assuming that the image sampling interval corresponding to image 1 determined based on the face detection result corresponding to image 1 is 2, then image 3 (1+2=3) can be extracted from the image sequence as the second frame candidate image, and image 3 can be input into the above face detection model. A face detection model is input, and the face detection result corresponding to image 3 outputted by the face detection model is obtained; assuming that the image sampling interval corresponding to image 3 determined based on the face detection result corresponding to image 3 is 1, image 4 (3+1=4) can be extracted from the image sequence as the third frame candidate image, image 4 is input into the above face detection model, and the face detection result corresponding to image 4 outputted by the face detection model is obtained; assuming that the image sampling interval corresponding to image 4 determined based on the face detection result corresponding to image 4 is 2, it means that image 4 is the last frame candidate image.

[0100] In summary, by iteratively executing the above image sampling detection process on the above image sequence, several frames of candidate images can be extracted from the image sequence. Moreover, in the process of extracting these several frames of candidate images from the image sequence, face detection of any one of these several frames of candidate images has been completed, and the corresponding face detection result has been obtained.

[0101] Step 302: Determine the candidate images whose face detection results meet the preset rules among the plurality of frames of candidate images as key images in the image sequence.

[0102] In this embodiment, when face detection results corresponding to the above-mentioned several frames of candidate images are obtained, the candidate images whose face detection results meet the preset rules among the several frames of candidate images can be determined as key images in the above-mentioned image sequence.

[0103] According to an implementation manner of the present disclosure, by iteratively executing an image sampling detection process, a candidate image whose face detection result meets a preset rule among several candidate frames extracted from an image sequence can be determined as a key image in the image sequence; and when executing the image sampling detection process, specifically based on the face detection result corresponding to the current frame candidate image output by the face detection model, the image sampling interval corresponding to the current frame candidate image can be determined, and based on the image interval, the next frame candidate image corresponding to the current frame candidate image can be extracted from the image sequence and input into the face detection model as a new current frame candidate image for face detection.

[0104] In this way, the number of images between the current frame candidate image and the next frame candidate image can be dynamically adjusted, thereby avoiding the need to calculate each frame image in the image sequence or multiple frames extracted at equal intervals, reducing the amount of calculation and improving the efficiency of key image extraction.

[0105] Furthermore, since the key image is determined by determining whether the face detection rule conforms to the preset rule, the extraction accuracy of the key image can be improved.

[0106] For a current frame candidate image in any image sampling detection process, two methods of determining an image sampling interval corresponding to the current frame candidate image based on a face detection result corresponding to the current frame candidate image are described below.

[0107] (1) The first method to determine the image sampling interval

[0108] In one embodiment shown, the face detection model performs face detection on any frame of image, on the one hand, to detect the number of faces contained in the frame of image, and on the other hand, to detect the key areas of each face in the frame of image. In this case, the face detection result corresponding to the frame of image output by the face detection model may include: the number of faces corresponding to the frame of image and the key areas of faces.

[0109] Correspondingly, when determining the image sampling interval corresponding to the current frame candidate image based on the face detection result corresponding to the above-mentioned current frame candidate image, the face state type corresponding to the current frame candidate image can be determined based on the number of faces corresponding to the current frame candidate image and the size of the face key area corresponding to the current frame candidate image. Subsequently, the image sampling interval corresponding to the face state type can be used as the image sampling interval corresponding to the current frame candidate image.

[0110] Furthermore, in one embodiment shown, since the shapes of a person's eyes and mouth usually change when a person makes an expression, the eye area and the mouth area can be used as the above-mentioned key areas of the face.

[0111] Furthermore, in an embodiment shown, the following method can be used to determine the face state type corresponding to the current frame candidate image based on the number of faces corresponding to the current frame candidate image and the size of the face key area corresponding to the current frame candidate image:

[0112] If the number of faces corresponding to the current frame candidate image is 0, it means that the current frame candidate image does not contain a face. At this time, the face state type corresponding to the current frame candidate image can be determined as the first type of face state.

[0113] If the number of faces corresponding to the current frame candidate image is greater than a preset threshold (which can be called the first threshold), it means that the current frame candidate image contains a large number of faces. At this time, the facial state type corresponding to the current frame candidate image can be determined as the second type of facial state.

[0114] If the number of faces corresponding to the current frame candidate image is greater than 0 and less than or equal to the above-mentioned first threshold (i.e., 0<the number of faces corresponding to the current frame candidate image ≤the first threshold), then the facial state type corresponding to the current frame candidate image can be further determined based on the size relationship between the size of the eye area corresponding to the current frame candidate image and a preset threshold (which may be called the second threshold), and the size relationship between the size of the mouth area corresponding to the current frame candidate image and a preset threshold (which may be called the third threshold).

[0115] Specifically, when the size of the eye region corresponding to the current frame candidate image is smaller than the second threshold, it can be determined that the facial state type corresponding to the current frame candidate image is the third type of facial state.

[0116] It is also possible to determine that the facial state type corresponding to the current frame candidate image is the fourth type of facial state when the size of the eye area corresponding to the current frame candidate image is greater than or equal to the second threshold, and the size of the mouth area corresponding to the current frame candidate image is greater than the third threshold.

[0117] It is also possible to determine that the facial state type corresponding to the current frame candidate image is the fifth type of facial state when the size of the eye area corresponding to the current frame candidate image is greater than or equal to the second threshold, and the size of the mouth area corresponding to the current frame candidate image is less than or equal to the third threshold.

[0118] In practical applications, the above face detection model can detect facial landmarks of any face in any frame of an image, and mark the detected facial landmarks on the image, so as to output the image with the facial landmarks marked. Facial landmarks are points used to locate the contour of the face, the facial features, etc., and are usually used as follows: Figure 4 The 49 facial key points shown are Figure 5 The 68 facial landmarks shown, and the 108 facial landmarks.

[0119] In addition, the above face detection model can also establish a rectangular coordinate system for any frame of image, and output the coordinates of the vertices of the rectangular area where any face in the image is located, as well as the coordinates of the facial key points corresponding to the face. Figure 6 As shown, the face detection model can output the coordinates of vertex 1, vertex 2, vertex 3, and vertex 4 of the rectangular area where face 1 is located, as well as the coordinates of the facial key points corresponding to face 1.

[0120] When determining the size of the key face region corresponding to the candidate image of the current frame, the size of the key face region can be calculated based on the coordinates of the vertices of the key face region. Since the size of the rectangular region where the face is located in different images is usually different, in order to adapt to different face proportions, the size of the key face region can be normalized based on the coordinates of the vertices of the rectangular region where the face is located.

[0121] Continue as Figure 6 As an example, assume that the coordinates of vertex 1 are (x1, y1), the coordinates of vertex 2 are (x2, y2), the coordinates of vertex 3 are (x3, y3), and the coordinates of vertex 4 are (x4, y4). Figure 5When face key point detection is performed on face 1 using the 68 face key points shown, the size R of the right eye area in face 1 can be calculated based on the coordinates of face key point No. 39 (assumed to be (xd39, yd39)) and face key point No. 41 (assumed to be (xd41, yd41)) related to the longitudinal length of the right eye area on face 1, as well as the coordinates of vertex 1 and vertex 2 (or vertex 3 and vertex 4), as shown in the following formula:

[0122]

[0123] Similarly, the size L of the left eye region in face 1 can be calculated based on the coordinates of face key point No. 44 (assumed to be (xd44, yd44)) and face key point No. 48 (assumed to be (xd48, yd48)) on face 1 that are related to the longitudinal length of the left eye region, and the coordinates of vertex 1 and vertex 2, as shown in the following formula:

[0124]

[0125] Similarly, the size M of the mouth area in face 1 can be calculated based on the coordinates of face key point No. 52 (assumed to be (xd52, yd52)) and face key point No. 58 (assumed to be (xd58, yd58)) related to the longitudinal length of the mouth area on face 1, as well as the coordinates of vertex 1 and vertex 2, as shown in the following formula:

[0126]

[0127] (2) The second method for determining the image sampling interval

[0128] In another embodiment shown, the face detection model performs face detection on any frame of image, and on the one hand, can detect the angle information corresponding to each face in the frame of image, on the other hand, can detect the appearance information corresponding to each face in the frame of image, and on the other hand, can detect the rectangular area (i.e., face area) where each face in the frame of image is located. In this case, the face detection result corresponding to the frame of image output by the face detection model may include: face angle information, face appearance information, and face area corresponding to the frame of image.

[0129] Correspondingly, when determining the image sampling interval corresponding to the current frame candidate image based on the face detection result corresponding to the above-mentioned current frame candidate image, the image quality score corresponding to the current frame candidate image can be calculated based on the preset scoring rules, as well as the face angle information and face appearance information corresponding to the current frame candidate image, and the position information of the face area corresponding to the current frame candidate image relative to the current frame candidate image. Subsequently, the image sampling interval corresponding to the image quality score can be used as the image sampling interval corresponding to the current frame candidate image.

[0130] Furthermore, in one embodiment shown, multiple image quality score intervals may be pre-set, and a corresponding image sampling interval may be set for each interval. In this case, when determining the image sampling interval corresponding to the image quality score of the above-mentioned current frame candidate image, the image quality score interval to which the image quality score corresponding to the current frame candidate image belongs may be determined based on the correspondence between the image quality score interval and the image sampling interval, and the image sampling interval corresponding to the image quality score interval may be determined as the image sampling interval corresponding to the image quality score of the current frame candidate image.

[0131] In practical applications, the angle information corresponding to the face is used to represent the three states of the face in Cartesian three-dimensional coordinates, namely pitch, yaw, and roll. Among them, pitch is the angle of up and down flipping, that is, the angle of rotation of the object around the X-axis; yaw is the angle of left and right flipping, that is, the angle of rotation of the object around the Y-axis; roll is the angle of rotation in the plane, that is, the angle of rotation of the object around the Z-axis. The above face detection model can perform angle detection on any face in any frame image, and output the pitch, yaw, and roll corresponding to the face as the face angle information. In this case, the score FA corresponding to the face angle information can be calculated as shown in the following formula:

[0132] FA=para pitch ×score pitch +para yaw ×SCore yaw +para roll ×sCore roll

[0133]

[0134]

[0135]

[0136] Among them, para pitch 、para yaw、para roll Represents the weights set for pitch, yaw, and roll respectively.

[0137] The above face detection model can also detect the appearance of any face in any frame image, and output the appearance score corresponding to the face, as well as the probability of detecting a face. In this case, the score FB corresponding to the face angle information can be calculated as shown in the following formula:

[0138]

[0139] Among them, FA represents the score corresponding to the face angle information, F beauty Indicates the appearance rating, F prob Indicates the probability of detecting a face, and eps represents a natural number greater than 0 (the specific value can be set according to the actual calculation situation, usually 0.000001).

[0140] That is, for any face, the score corresponding to the facial appearance information of the face can be calculated based on the score corresponding to the facial angle information of the face.

[0141] The above face detection model can also establish a rectangular coordinate system for any frame of image and output the coordinates of the vertices of the rectangular area where any face in the image is located. In this case, the score FL corresponding to the position information of the face area relative to the image can be calculated as shown in the following formula:

[0142]

[0143] Among them, (c x0 , c y0 ) is the coordinate of the center position of the image, (c x1 , c y1 ) is the coordinate of the center position of the face area.

[0144] Continue as Figure 6 Taking the image shown as an example, assuming that a rectangular coordinate system is established with the lower left vertex of the image as the origin, the side of the rectangular area where face 1 is located is parallel to the x-axis or y-axis, w represents the horizontal width of the image in the rectangular coordinate system, h represents the vertical length of the image in the rectangular coordinate system, the coordinates of vertex 1 are (x1, y1), the coordinates of vertex 2 are (x2, y2), the coordinates of vertex 3 are (x3, y3), and the coordinates of vertex 4 are (x4, y4), then the coordinates of the center position of the image and the coordinates of the center position of the rectangular area where face 1 is located can be calculated, as shown in the following formula:

[0145]

[0146]

[0147]

[0148]

[0149] In practical applications, for any frame of image, if there are multiple faces in it, the average of the scores corresponding to the facial angle information of these multiple faces can be used as the score corresponding to the facial angle information of the image; the average of the scores corresponding to the facial appearance information of these multiple faces can be used as the score corresponding to the facial appearance information of the image; the average of the scores corresponding to the position information of these multiple faces relative to the image can be used as the score corresponding to the position information of the image.

[0150] For any frame of image, after calculating the scores FA, FB, and FL corresponding to its face angle information, face value information, and position information, the image quality score F corresponding to the image can be further calculated. Since different images are usually of different sizes, in order to adapt to different image sizes, the image quality score F can be normalized based on the coordinates of the center position of the image, as shown in the following formula:

[0151]

[0152] For any image sequence, three methods are described below for determining candidate images whose face detection results meet preset rules from a number of candidate image frames extracted from the image sequence as key images in the image sequence.

[0153] (1) The first method of determining the key image

[0154] In one embodiment shown, when the above-mentioned face detection results include the number of faces and the key face areas, the candidate images whose number of faces and the sizes of the key face areas meet the preset rules among several frames of candidate images extracted from the above-mentioned image sequence can be determined as the key images in the image sequence.

[0155] It should be noted that the calculation of the size of the key area of ​​the face can refer to the aforementioned related content, and the present disclosure will not go into details here.

[0156] In practical applications, the preset rule may include: the size of the eye area is greater than or equal to the second threshold, and the size of the mouth area is less than or equal to the third threshold. That is, the candidate image with the face state type of the fifth type of face state can be determined as the key image in the image sequence.

[0157] (2) The second method of determining the key image

[0158] In another embodiment shown, when the above-mentioned face detection result includes face angle information, face appearance information and face area, the image quality score corresponding to any candidate image can be calculated based on the preset scoring rules, as well as the face angle information, face appearance information and the position information of the face area relative to the candidate image, so that the candidate image with an image quality score greater than or equal to a preset threshold (which may be called the fourth threshold) can be determined as the key image in the image sequence. Alternatively, all candidate images can be sorted in order of image quality scores from large to small, and a preset number of candidate images in the front can be determined as the key images in the image sequence.

[0159] It should be noted that the calculation of the image quality score may refer to the aforementioned related content, and the present disclosure will not go into details here.

[0160] (3) The third method of determining the key image

[0161] In another embodiment shown, when the above face detection result includes the number of faces, key face areas, face angle information, face appearance information and face area, the candidate images whose number of faces and the size of key face areas meet the preset rules among the several frames of candidate images extracted from the above image sequence can be determined as target images, and further, for any target image, based on the preset scoring rules, as well as the face angle information, face appearance information and the position information of the face area relative to the target image, the image quality score corresponding to the target image is calculated, so that the target image with an image quality score greater than or equal to the fourth threshold can be determined as the key image in the image sequence. Alternatively, all target images can be sorted in order of image quality scores from large to small, and a preset number of target images at the front can be determined as the key images in the image sequence.

[0162] It should be noted that the calculation of the size of the key face area and the calculation of the image quality score can refer to the aforementioned related content, and the present disclosure will not go into details here.

[0163] Exemplary Media

[0164] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Figure 7 A medium for extracting a key image in an image sequence according to an exemplary embodiment of the present disclosure is described.

[0165] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus, or a device.

[0166] The program product may adopt any combination of one or more readable media, which may be readable signal media or readable storage media.

[0167] The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0168] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0169] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0170] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0171] Exemplary Devices

[0172] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Figure 8 A device for extracting a key image from an image sequence according to an exemplary embodiment of the present disclosure is described.

[0173] The implementation process of the functions and effects of each module in the following device is specifically referred to the implementation process of the corresponding steps in the above method, and the present disclosure will not repeat them here. For the device embodiment, it basically corresponds to the method embodiment, so the relevant parts can be referred to the partial description of the method embodiment.

[0174] refer to Figure 8 , Figure 8 A device for extracting key images from an image sequence according to an embodiment of the present disclosure is schematically shown.

[0175] The device for extracting key images from the above image sequence may include:

[0176] The detection module 801 is used to iteratively execute the image sampling detection process to extract a plurality of candidate image frames from the image sequence, and perform face detection on the plurality of candidate image frames;

[0177] A determination module 802 is used to determine the candidate images whose face detection results meet the preset rules among the plurality of frames of candidate images as key images in the image sequence;

[0178] Wherein, the detection module 801 includes:

[0179] The acquisition submodule 8011 is used to obtain the face detection result output by the face detection model corresponding to the candidate image of the current frame;

[0180] A determination submodule 8012 is used to determine an image sampling interval corresponding to the candidate image of the current frame based on the face detection result;

[0181] The extraction submodule 8013 is used to extract the next frame candidate image corresponding to the current frame candidate image from the image sequence; wherein the number of images between the next frame candidate image and the current frame candidate image in the image sequence matches the image sampling interval;

[0182] The detection submodule 8014 is used to input the next frame candidate image as a new current frame candidate image into the face detection model, so that the face detection model performs face detection on the next frame candidate image.

[0183] Optionally, the image sequence is a video;

[0184] The determining submodule 8012 is specifically used for:

[0185] Determining an image sampling rate corresponding to the face detection result;

[0186] The product of the image sampling rate and the frame rate of the video is determined as the image sampling interval corresponding to the current frame candidate image.

[0187] Optionally, the face detection result includes the number of faces and key face areas;

[0188] The determining submodule 8012 is specifically used for:

[0189] Determining the type of face state corresponding to the candidate image of the current frame based on the number of faces and the size of the key face area;

[0190] Determine an image sampling interval corresponding to the face state type.

[0191] Optionally, the key facial areas include an eye area and a mouth area.

[0192] Optionally, the determining submodule 8012 is specifically used for:

[0193] If the number of faces is 0, determining that the face state type corresponding to the candidate image of the current frame is the first type of face state;

[0194] If the number of faces is greater than a preset first threshold, determining that the face state type corresponding to the candidate image of the current frame is a second type of face state;

[0195] If the number of faces is greater than 0 and less than or equal to the first threshold, the facial state type corresponding to the current frame candidate image is determined based on the size relationship between the size of the eye area and a preset second threshold, and the size relationship between the size of the mouth area and a preset third threshold.

[0196] Optionally, the determining submodule 8012 is specifically used for:

[0197] If the size of the eye area is smaller than a preset second threshold, determining that the facial state type corresponding to the current frame candidate image is a third type of facial state;

[0198] If the size of the eye area is greater than or equal to the second threshold, and the size of the mouth area is greater than a preset third threshold, determining that the facial state type corresponding to the candidate image of the current frame is a fourth facial state;

[0199] If the size of the eye area is greater than or equal to the second threshold, and the size of the mouth area is less than or equal to the third threshold, it is determined that the facial state type corresponding to the current frame candidate image is the fifth type of facial state.

[0200] Optionally, the face detection result includes face angle information, face appearance information and face area;

[0201] The determining submodule 8012 is specifically used for:

[0202] Calculate the image quality score corresponding to the candidate image of the current frame based on a preset scoring rule, the face angle information, the face appearance information, and the position information of the face region relative to the candidate image of the current frame;

[0203] An image sampling interval corresponding to the image quality score is determined.

[0204] Optionally, the determining submodule 8012 is specifically used for:

[0205] Based on the correspondence between the preset image quality score interval and the image sampling interval, the image quality score interval to which the image quality score belongs is determined, and the image sampling interval corresponding to the image quality score interval is determined as the image sampling interval corresponding to the image quality score.

[0206] Optionally, the face detection result includes the number of faces and key face areas;

[0207] The determining module 802 is specifically used for:

[0208] Among the plurality of frames of candidate images, the candidate images whose number of faces and sizes of key face regions meet preset rules are determined as key images in the image sequence.

[0209] Optionally, the face detection result also includes face angle information, face appearance information and face area;

[0210] The determining module 802 is specifically used for:

[0211] Determine, as a target image, a candidate image whose number of faces and the size of the key face area meet preset rules among the candidate images of the plurality of frames;

[0212] For any target image, based on a preset scoring rule, as well as the face angle information, the face appearance information, and the position information of the face area relative to the target image, calculate an image quality score corresponding to the target image;

[0213] The target image whose image quality score is greater than or equal to a preset fourth threshold is determined as a key image in the image sequence.

[0214] Optionally, the face detection result includes face angle information, face appearance information and face area;

[0215] The determining module 802 is specifically used for:

[0216] For any candidate image, based on a preset scoring rule, as well as the face angle information, the face appearance information, and the position information of the face area relative to the candidate image, calculate an image quality score corresponding to the candidate image;

[0217] Among the plurality of candidate image frames, the candidate images whose image quality scores are greater than or equal to a preset fifth threshold are determined as key images in the image sequence.

[0218] Exemplary Computing Devices

[0219] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Fig. 9 A computing device for extracting key images from an image sequence according to an exemplary embodiment of the present disclosure is described.

[0220] Fig. 9 The computing device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0221] like Fig. 9 As shown, the computing device 900 is in the form of a general computing device. The components of the computing device 900 may include but are not limited to: at least one processing unit 901, at least one storage unit 902, and a bus 903 connecting different system components (including the processing unit 901 and the storage unit 902).

[0222] The bus 903 includes a data bus, a control bus, and an address bus.

[0223] The storage unit 902 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 9021 and / or a cache memory 9022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 9023 .

[0224] The storage unit 902 may also include a program / utility 9025 having a set (at least one) of program modules 9024, such program modules 9024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0225] The computing device 900 may also communicate with one or more external devices 904 (eg, keyboards, pointing devices, etc.).

[0226] Such communication may be performed via input / output (I / O) interface 905. Furthermore, computing device 900 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via network adapter 906. Fig. 9 As shown, the network adapter 906 communicates with other modules of the computing device 900 via the bus 903. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0227] It should be noted that, although several units / modules or sub-units / modules of the device for extracting key images in an image sequence are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided into multiple units / modules for embodiment.

[0228] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0229] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for extracting a key image from an image sequence, the method comprising: Iteratively executing the image sampling detection process to extract a number of candidate image frames from the image sequence, and performing face detection on the number of candidate image frames; Determine the candidate images whose face detection results meet the preset rules among the plurality of frames of candidate images as key images in the image sequence; The image sampling detection process includes: Obtain the face detection result output by the face detection model corresponding to the candidate image of the current frame; Based on the face detection result, determining an image sampling interval corresponding to the current frame candidate image; Extracting a next frame candidate image corresponding to the current frame candidate image from the image sequence; wherein the number of images between the next frame candidate image and the current frame candidate image in the image sequence matches the image sampling interval; The next frame candidate image is used as a new current frame candidate image and input into the face detection model, so that the face detection model performs face detection on the next frame candidate image.

2. The method according to claim 1, wherein the image sequence is a video; The step of determining the image sampling interval corresponding to the current frame candidate image based on the face detection result includes: Determining an image sampling rate corresponding to the face detection result; The product of the image sampling rate and the frame rate of the video is determined as the image sampling interval corresponding to the current frame candidate image.

3. The method according to claim 1, wherein the face detection result includes the number of faces and key face areas; The step of determining the image sampling interval corresponding to the current frame candidate image based on the face detection result includes: Determining the type of face state corresponding to the candidate image of the current frame based on the number of faces and the size of the key face area; Determine an image sampling interval corresponding to the face state type.

4. According to the method of claim 1, the face detection result includes face angle information, face appearance information and face area; The step of determining the image sampling interval corresponding to the current frame candidate image based on the face detection result includes: Calculate the image quality score corresponding to the candidate image of the current frame based on a preset scoring rule, the face angle information, the face appearance information, and the position information of the face region relative to the candidate image of the current frame; An image sampling interval corresponding to the image quality score is determined.

5. The method according to claim 1, wherein the face detection result includes the number of faces and key face areas; The step of determining the candidate images whose face detection results meet a preset rule among the plurality of frames of candidate images as key images in the image sequence comprises: Among the plurality of frames of candidate images, the candidate images whose number of faces and sizes of key face regions meet preset rules are determined as key images in the image sequence.

6. The method according to claim 5, wherein the face detection result further includes face angle information, face appearance information and face area; The step of determining, among the plurality of candidate images, the candidate images whose number of faces and sizes of the key face regions meet preset rules as key images in the image sequence comprises: Determine, as a target image, a candidate image whose number of faces and the size of the key face area meet preset rules among the candidate images of the plurality of frames; For any target image, based on a preset scoring rule, as well as the face angle information, the face appearance information, and the position information of the face area relative to the target image, calculate an image quality score corresponding to the target image; The target image whose image quality score is greater than or equal to a preset fourth threshold is determined as a key image in the image sequence.

7. The method according to claim 1, wherein the face detection result includes face angle information, face appearance information and face area; The step of determining the candidate images whose face detection results meet a preset rule among the plurality of frames of candidate images as key images in the image sequence comprises: For any candidate image, based on a preset scoring rule, as well as the face angle information, the face appearance information, and the position information of the face area relative to the candidate image, calculate an image quality score corresponding to the candidate image; Among the plurality of candidate image frames, the candidate images whose image quality scores are greater than or equal to a preset fifth threshold are determined as key images in the image sequence.

8. A device for extracting key images from an image sequence, the device comprising: A detection module, used for iteratively executing an image sampling detection process to extract a number of candidate image frames from an image sequence, and performing face detection on the number of candidate image frames; A determination module, used to determine the candidate images whose face detection results meet the preset rules among the plurality of frames of candidate images as key images in the image sequence; Wherein, the detection module includes: The acquisition submodule is used to obtain the face detection result output by the face detection model corresponding to the candidate image of the current frame; A determination submodule, configured to determine an image sampling interval corresponding to the candidate image of the current frame based on the face detection result; An extraction submodule, configured to extract a next frame candidate image corresponding to the current frame candidate image from the image sequence; wherein the number of images between the next frame candidate image and the current frame candidate image in the image sequence matches the image sampling interval; The detection submodule is used to input the next frame candidate image as a new current frame candidate image into the face detection model so that the face detection model performs face detection on the next frame candidate image.

9. A medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. A computing device comprising: processor; a memory for storing a processor executable program; The processor implements the method according to any one of claims 1 to 7 by running the executable program.

Images