Dynamic cover generation method and apparatus, and electronic device

Abstract

The application discloses a dynamic cover generation method and device and electronic equipment. Frame extraction is performed on a target video to obtain a video picture set; human body key point detection is performed on each frame of video picture in the video picture set; for each frame of video picture in the video picture set, a picture score of the video picture is determined based on a key point detection result of the video picture; a plurality of candidate picture sets are constructed based on the video picture set, a comprehensive score of each candidate picture set is determined, and a first number of target picture sets are selected from the plurality of candidate picture sets at least according to a principle of high priority of the comprehensive score; and a dynamic cover of the target video is generated based on the first number of target picture sets. The technical scheme disclosed by the application automatically generates a dynamic cover for a target video without human participation, has high efficiency, consumes very short time, does not miss wonderful segments, and the generated dynamic cover contains more human interaction pictures, which is conducive to attracting the attention of users.

Description

Dynamic Cover Generation Method, Apparatus and Electronic Equipment Technical Field

[0001] This application belongs to the field of multimedia technology, and in particular relates to a method, apparatus and electronic device for generating dynamic covers. Background Technology

[0002] With the development of information technology and the widespread adoption of smart devices, more and more video applications have emerged, allowing users to watch videos through these applications installed on their smart devices. To help users quickly understand the content of each video, video platforms display a cover image. An attractive cover image can often capture users' attention, thus garnering more attention for the video.

[0003] Compared to static video covers, animated covers can display more information from a video, thereby attracting more user attention and achieving higher click-through rates. Currently, the generation of animated covers relies on manual processes: staff browse videos, select compelling clips, and then use these selected clips to generate animated video covers.

[0004] As can be seen, existing dynamic cover generation solutions rely on manual implementation, which suffers from low efficiency and long processing time. In addition, the manual selection process is prone to missing exciting clips. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a dynamic cover generation method, apparatus and electronic device to solve the problem that existing dynamic cover generation schemes rely on manual implementation. Moreover, in the process of automatically generating dynamic covers for videos, the degree of human interaction in the video is used as the basis, which makes the generated dynamic covers contain more human interaction scenes, which is conducive to attracting users' attention.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] On the one hand, this application provides a method for generating a dynamic cover, including:

[0008] Frames are extracted from the target video to obtain a set of video images;

[0009] Human key point detection is performed on each frame of the video image set;

[0010] For each frame of video image in the video image set, an image score is determined based on the key point detection results of the video image. When the number of human bodies contained in the video image exceeds a specified range, the image score of the video image is a first value. When the number of human bodies contained in the video image is within the specified range, the image score of the video image is positively correlated with the degree of interaction of the human bodies in the video image.

[0011] Based on the video image set, multiple candidate image sets are constructed, and the comprehensive score of each candidate image set is determined. At least a first number of target image sets are selected from the multiple candidate image sets based on the principle of prioritizing those with higher comprehensive scores. The comprehensive score of a candidate image set is determined by the image scores of the video images contained in the candidate image set.

[0012] Based on the first set of target images, a dynamic cover is generated for the target video.

[0013] Optionally, determining the image score of the video image based on the key point detection results includes:

[0014] The number of human bodies contained in the video image is determined based on the key point detection results.

[0015] If the number of human bodies contained in the video image exceeds the specified range, then the image score of the video image is determined to be a first value;

[0016] If the number of human bodies contained in the video image is within the specified range, then for each human body in the video image, a limb distance score and a face distance score of that human body relative to each other human body are determined. Based on the limb distance scores and face distance scores corresponding to all human bodies contained in the video image, an image score of the video image is determined.

[0017] Optionally, for any video image, determine the limb distance score of the first human body relative to the second human body, including:

[0018] The distance scores of the limbs of the first human body relative to the second human body are determined according to the formula arm_dis_12=1 / min_diss(P_1[arm_index],P_2[all]);

[0019] For any given video image, determine the facial distance score between the first and second human figures within it, including:

[0020] The face distance score between the first human body and the second human body is determined according to the formula face_dis_12=1 / min_diss(P_1[face_index],P_2[all]);

[0021] Wherein, arm_dis_12 is the limb distance score of the first human body relative to the second human body, face_dis_12 is the face distance score of the first human body relative to the second human body, arm_index is the index of the limb keypoint, P_1[arm_index] is the coordinate of the limb keypoint of the first human body, face_index is the index of the face keypoint, P_1[face_index] is the coordinate of the face keypoint of the first human body, P_2[all] is the coordinate of all human body keypoints of the second human body, min_diss(P_1[arm_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the limb keypoints of the first human body and all human body keypoints of the second human body, and min_diss(P_1[face_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the face keypoints of the first human body and all human body keypoints of the second human body.

[0022] Optionally, for any video image, determine the limb distance score of the first human body relative to the second human body, including:

[0023] The distance scores of the limbs of the first human body relative to the second human body are determined according to the formula arm_dis_12=area_ratio / min_diss(P_1[arm_index],P_2[all]);

[0024] For any given video image, determine the facial distance score between the first and second human figures within it, including:

[0025] The face distance score of the first human body relative to the second human body is determined according to the formula face_dis_12=area_ratio / min_diss(P_1[face_index],P_2[all]);

[0026] Wherein, arm_dis_12 is the limb distance score of the first human body relative to the second human body, face_dis_12 is the face distance score of the first human body relative to the second human body, arm_index is the index of the limb keypoint, P_1[arm_index] is the coordinate of the limb keypoint of the first human body, face_index is the index of the face keypoint, P_1[face_index] is the coordinate of the face keypoint of the first human body, P_2[all] is the coordinate of all human body keypoints of the second human body, min_diss(P_1[arm_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the limb keypoints of the first human body and all human body keypoints of the second human body, min_diss(P_1[face_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the face keypoints of the first human body and all human body keypoints of the second human body, and area_ratio is the balance coefficient.

[0027] Optionally, the balance coefficient is:

[0028]

[0029] Where area_ratio is the balance coefficient, box_area(P_1) is the area of ​​the smallest rectangle containing all the key points of the first human body, box_area(P_2) is the area of ​​the smallest rectangle containing all the key points of the second human body, img_area is the area of ​​the video image, min() is to take the minimum value, and max() is to take the maximum value.

[0030] Optionally, determining the image score of the video image based on the limb distance scores and face distance scores corresponding to all human bodies contained in the video image includes:

[0031] Find the maximum value among the limb distance scores of all human figures contained in the video images;

[0032] Find the maximum value among the face distance scores corresponding to all human bodies contained in the video image;

[0033] The sum of the two maximum values ​​found is used as the image score for the video image.

[0034] Optionally, the construction of multiple alternative image sets based on the video image set includes:

[0035] Starting with each frame of the video image set, a second number of video images are obtained from the video image set to form multiple candidate image sets.

[0036] Optionally, selecting a first number of target images from the plurality of candidate image sets based on the principle of prioritizing higher overall scores includes:

[0037] Based on the overall score of all candidate image sets, the NMS algorithm is used to select the first number of target image sets from the multiple candidate image sets.

[0038] On the other hand, this application provides a dynamic cover generation device, comprising:

[0039] The video preprocessing module is used to extract frames from the target video to obtain a video image set;

[0040] The key point detection module is used to perform human key point detection on each frame of the video image set.

[0041] The image score determination module is used to determine the image score of each video image in the video image set based on the key point detection results of the video image. When the number of human bodies contained in the video image exceeds a specified range, the image score of the video image is 0. When the number of human bodies contained in the video image is within the specified range, the image score of the video image is positively correlated with the degree of interaction of the human bodies in the video image.

[0042] The target image set generation module is used to construct multiple candidate image sets based on the video image set, determine the comprehensive score of each candidate image set, and select a first number of target image sets from the multiple candidate image sets based on the principle of selecting the highest comprehensive score. The comprehensive score of the candidate image set is determined by the image score of the video images contained in the candidate image set.

[0043] The dynamic cover generation module is used to generate a dynamic cover for the target video based on the first number of target image sets.

[0044] On the other hand, this application provides an electronic device, including a processor and a memory;

[0045] The memory is used to store programs;

[0046] The processor is used to execute the program and implement each step of any of the above methods.

[0047] Therefore, the beneficial effects of this application are as follows:

[0048] The dynamic cover generation method disclosed in this application first extracts frames from the target video to obtain a video image set. Then, for each video image in the video image set, human keypoint detection is performed on the video image, and the image score of the video image is determined based on the keypoint detection results. When the number of human figures contained in the video image exceeds a specified range, the image score of the video image is a first value (e.g., 0). When the number of human figures contained in the video image is within the specified range, the image score of the video image is positively correlated with the degree of interaction of the human figures in the video image. Next, multiple candidate image sets are constructed based on the video image set, and the comprehensive score of each candidate image set is determined (determined by the image scores of the video images contained in the candidate image set). At least based on the principle of prioritizing higher comprehensive scores, a first number of target image sets are selected from the multiple candidate image sets. Finally, a dynamic cover of the target video is generated based on the first number of candidate image sets. As can be seen, the dynamic cover generation method disclosed in this application automatically generates dynamic covers for target videos without manual intervention, which is highly efficient, takes very little time, and does not miss any exciting parts. In addition, the image score can intuitively reflect the degree of human interaction in the video images. Subsequently, the highest comprehensive score is prioritized to select the first number of target images from multiple candidate image sets. This results in the selected target image set containing more human interaction scenes, thus making the generated dynamic cover contain more human interaction scenes, which is conducive to attracting users' attention. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 is a flowchart of a dynamic cover generation method disclosed in this application;

[0051] Figure 2 is a flowchart of the method for determining the image score of a video image disclosed in this application;

[0052] Figure 3 is a structural schematic diagram of a dynamic cover generation device disclosed in this application;

[0053] Figure 4 is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0054] This application discloses a dynamic cover generation method and apparatus, which can solve the problem that existing dynamic cover generation schemes rely on manual implementation. Moreover, in the process of automatically generating dynamic covers for videos, the degree of human interaction in the video is used as the basis, which makes the generated dynamic covers contain more human interaction scenes, which is conducive to attracting users' attention.

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] Referring to Figure 1, Figure 1 is a flowchart of a dynamic cover generation method disclosed in this application. This method is executed by an electronic device, which can be a server or other electronic devices with computing capabilities. The method includes:

[0057] S101: Extract frames from the target video to obtain a video image set.

[0058] The target video is the video for which an animated cover needs to be generated.

[0059] Optionally, in the target video, one video image is extracted at regular intervals to form a video image set. For example, one video image is extracted every 5 frames to form a video image set.

[0060] Optionally, the video images in the video image set are processed to a predetermined size. To ensure high processing efficiency when the electronic device generates dynamic covers for different target videos, the video images in the video image set are adjusted to the same size. For example, the video images in the video image set are processed to have a width of 800 pixels and a height of 600 pixels. Of course, this is just an example regarding the size of video images, and this application is not limited to this.

[0061] S102: Perform human keypoint detection on each frame of the video image set.

[0062] In practice, existing algorithms or models are used to detect human keypoints in each frame of the video image set to obtain the keypoints of the human body in each frame. For example, the DEKR model can be used to detect human keypoints in video images.

[0063] This section explains the key points of the human body.

[0064] Optionally, there are 17 key points for the human body, including 5 facial key points and 12 limb key points. The 5 facial key points are: nose, left eye, right eye, left ear, and right ear. The 12 limb key points are: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0065] In practice, more human body key points can be set as needed, or some of the aforementioned human body key points can be omitted as needed.

[0066] S103: For each frame of video image in the video image set, determine the image score of the video image based on the key point detection results of the video image.

[0067] Specifically, when the number of human figures contained in a video image exceeds a specified range, the image score of the video image is the first value; when the number of human figures contained in a video image is within a specified range, the image score of the video image is positively correlated with the degree of interaction of the human figures in the video image.

[0068] In practice, the first value is preferably 0. Of course, the first value can also be a positive number close to 0. For example, the first value is a value between 0 and 0.1; for example, the first value is a value between 0 and 0.01. Those skilled in the art can adjust the upper limit of the first value. Of course, the closer the upper limit of the first value is to 0, the better.

[0069] The applicant discovered that animated covers, compared to static covers, contain more information and are more effective at attracting user attention. Furthermore, animated covers that include human interaction, such as hugging, kissing, holding hands, pushing, or fighting, are even more effective at capturing user attention. Therefore, this application uses the degree of human interaction as a basis in the process of generating animated covers for videos, ensuring that the generated animated covers contain more human interaction to attract user attention.

[0070] Based on the aforementioned technical concept, the degree of human interaction in video images is used as the basis for determining the image score of a video image. It's understandable that if a video image contains no human body, or only one human body, then there is no human interaction. Furthermore, if a video image contains too many human bodies, the resulting effect is cluttered and is generally not ideal material for creating dynamic cover images.

[0071] In this application, for each frame of a video image in the video image set, if the number of human bodies contained in the video image exceeds a specified range, then the image score of the video image is determined as the first value; if the number of human bodies contained in the video image is within the specified range, then the image score of the video image is positively correlated with the degree of interaction of the human bodies in the video image, that is, the higher the degree of interaction of the human bodies in the video image, the higher the image score of the video image.

[0072] Optionally, the specified range is 2 to 3. That is, if a video image contains 2 or 3 human figures, then the image score of the video image is positively correlated with the degree of interaction of the human figures in the video image; if a video image does not contain any human figures, contains 1 human figure, or contains more than 3 human figures, then the image score of the video image is determined to be the first value. Of course, the above is only an example of the specified range, and this application is not limited to this.

[0073] S104: Construct multiple candidate image sets based on the video image set, determine the comprehensive score of each candidate image set, and select the first number of target image sets from the multiple candidate image sets based on the principle of prioritizing those with higher comprehensive scores.

[0074] The overall score of the candidate image set is determined by the image scores of the video images contained in the candidate image set.

[0075] Optionally, the overall score of the candidate image set is the sum of the image scores of the video images contained in the candidate image set.

[0076] It should be noted that the selection of the first set of target images from multiple candidate image sets should be based on the principle of selecting the highest overall score. Of course, the method for selecting the first set of target images from multiple candidate image sets can be further optimized.

[0077] S105: Generate a dynamic cover for the target video based on a first set of target images.

[0078] An animated cover can be understood as a GIF image. GIF (Graphics Interchange Format) is a common image file format standard. Its principle is to save multiple frames of images and then display multiple frames of images in succession to produce a dynamic effect.

[0079] In practice, a first set of target images is saved, and the image playback speed is set (e.g., 24fps, meaning 24 frames per second). This generates the dynamic cover for the target video. Subsequently, simply play the images from the target image set one by one at the set playback speed to produce the dynamic effect.

[0080] The dynamic cover generation method disclosed in this application first extracts frames from the target video to obtain a video image set. Then, for each video image in the video image set, human keypoint detection is performed on the video image, and the image score of the video image is determined based on the keypoint detection results. When the number of human figures contained in the video image exceeds a specified range, the image score of the video image is a first value (e.g., 0). When the number of human figures contained in the video image is within the specified range, the image score of the video image is positively correlated with the degree of interaction of the human figures in the video image. Next, multiple candidate image sets are constructed based on the video image set, and the comprehensive score of each candidate image set is determined (determined by the image scores of the video images contained in the candidate image set). At least based on the principle of prioritizing higher comprehensive scores, a first number of target image sets are selected from the multiple candidate image sets. Finally, a dynamic cover of the target video is generated based on the first number of candidate image sets. As can be seen, the dynamic cover generation method disclosed in this application automatically generates dynamic covers for target videos without manual intervention, which is highly efficient, takes very little time, and does not miss any exciting parts. In addition, the image score can intuitively reflect the degree of human interaction in the video images. Subsequently, the highest comprehensive score is prioritized to select the first number of target images from multiple candidate image sets. This results in the selected target image set containing more human interaction scenes, thus making the generated dynamic cover contain more human interaction scenes, which is conducive to attracting users' attention.

[0081] In another embodiment of this application, the focus is on describing a scheme for constructing multiple alternative image sets based on video image sets.

[0082] Multiple candidate image sets are constructed based on the video image set, including: taking each frame of video image in the video image set as the starting point, obtaining a second number of video images from the video image set to form multiple candidate image sets.

[0083] Considering the visual effect of dynamic covers, each set of alternative images can be set to contain 41 video images, i.e., the second quantity is 41. Of course, this is just one example regarding the second quantity, and this application is not limited to it.

[0084] This section uses a second set of 41 images as an example to illustrate the scheme of constructing multiple candidate image sets based on video image sets. The video images in the video image set are extracted from the target video. According to the order in which the video images appear in the target video, they are sequentially labeled as frame 1, frame 2, frame 3, ..., frame N. In practical applications, video images are typically extracted from the beginning to the end of the target video at preset intervals. The first extracted video image is stored in the video image set first; therefore, the frame numbers of the video images in the video image set can also be marked according to the order in which they are stored.

[0085] Starting with the first frame of the video image set, obtain 41 consecutive video frames to form a candidate image set; starting with the second frame, obtain another 41 consecutive video frames to form a candidate image set; and so on, constructing multiple candidate image sets. It should be noted that if, starting from frame N-39 of the video image set, it is impossible to obtain 41 consecutive video frames, the candidate image set containing fewer than the second-largest number of video images can be discarded.

[0086] In another embodiment of this application, the scheme of selecting a first number of target images from multiple candidate image sets is described in particular, based on the principle of prioritizing those with higher overall scores.

[0087] At least one target image set should be selected from multiple candidate image sets based on the principle of prioritizing those with higher overall scores. This includes selecting the first number of target image sets from multiple candidate image sets using the NMS algorithm based on the overall scores of all candidate image sets.

[0088] NMS stands for Non-Maximum Suppression. Its principle is to suppress elements that are not local maxima and search for local maxima. The process of selecting the first set of target images from multiple candidate image sets using the NMS algorithm can be understood as follows: if two candidate image sets intersect (or overlap, i.e., they contain the same video images), then the candidate image set with the higher overall score is retained. The NMS algorithm selects the first set of target images from multiple candidate image sets that satisfy the following conditions: they do not overlap and have the highest overall score.

[0089] In another embodiment of this application, we focus on introducing a scheme for determining the image score of a video image based on the key point detection results of the video image.

[0090] Referring to Figure 2, which is a flowchart of the method for determining the image score of a video image disclosed in this application, the method includes:

[0091] S201: Determine the number of human bodies contained in the video image based on the key point detection results.

[0092] S202: If the number of human bodies contained in a video image exceeds the specified range, then the image score of the video image is determined to be the first value.

[0093] S203: If the number of human bodies contained in the video image is within a specified range, then for each human body in the video image, determine the limb distance score and face distance score of the human body relative to each other human body.

[0094] For example, if a video frame contains two human bodies, referred to as human body A and human body B, then it is necessary to determine: the limb distance score of human body A relative to human body B, the face distance score of human body A relative to human body B, the limb distance score of human body B relative to human body A, and the face distance score of human body B relative to human body A.

[0095] For example, if a video frame contains three human bodies, referred to as human body A, human body B, and human body C, then it is necessary to determine: the limb distance score of human body A relative to human body B, the face distance score of human body A relative to human body B, the limb distance score of human body A relative to human body C, the face distance score of human body A relative to human body C, the limb distance score of human body B relative to human body A, the face distance score of human body B relative to human body C, the limb distance score of human body C relative to human body A, the face distance score of human body C relative to human body A, the limb distance score of human body C relative to human body B, and the face distance score of human body C relative to human body B.

[0096] In practice, various methods can be used to determine the limb distance score and face distance score of one human body relative to another human body in a video image.

[0097] First option:

[0098] For any video image, determine the limb distance score of the first human body relative to the second human body, including: determining the limb distance score of the first human body relative to the second human body according to formula (1).

[0099] arm_dis_12=1 / min_diss(P_1[arm_index],P_2[all] Formula (1)

[0100] For any video image, determine the facial distance score between the first human body and the second human body, including: determining the facial distance score between the first human body and the second human body according to formula (2).

[0101] face_dis_12=1 / min_diss(P_1[face_index],P_2[all] Formula (2)

[0102] Wherein, arm_dis_12 is the limb distance score of the first human body relative to the second human body, face_dis_12 is the face distance score of the first human body relative to the second human body, arm_index is the index of the limb keypoint, P_1[arm_index] is the coordinate of the limb keypoint of the first human body, face_index is the index of the face keypoint, P_1[face_index] is the coordinate of the face keypoint of the first human body, P_2[all] is the coordinate of all human body keypoints of the second human body, min_diss(P_1[arm_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the limb keypoints of the first human body and all human body keypoints of the second human body, and min_diss(P_1[face_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the face keypoints of the first human body and all human body keypoints of the second human body.

[0103] As can be seen, in any frame of video image, the closer the limbs of the first human are to the second human, the higher the distance score of the first human relative to the second human's limbs; the closer the face of the first human is to the second human, the higher the distance score of the first human relative to the second human's face.

[0104] The second option:

[0105] For any video image, determine the limb distance score of the first human body relative to the second human body, including: determining the limb distance score of the first human body relative to the second human body according to formula (3).

[0106] arm_dis_12=area_ratio / min_diss(P_1[arm_index],P_2[all] Formula (3)

[0107] For any video image, determine the facial distance score between the first human body and the second human body, including: determining the facial distance score between the first human body and the second human body according to formula (4).

[0108] face_dis_12=area_ratio / min_diss(P_1[face_index],P_2[all] Formula (4)

[0109] Wherein, arm_dis_12 is the limb distance score of the first human body relative to the second human body, face_dis_12 is the face distance score of the first human body relative to the second human body, arm_index is the index of the limb keypoints, P_1[arm_index] is the coordinate of the limb keypoint of the first human body, face_index is the index of the face keypoint, P_1[face_index] is the coordinate of the face keypoint of the first human body, P_2[all] is the coordinate of all human body keypoints of the second human body, min_diss(P_1[arm_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the limb keypoints of the first human body and all human body keypoints of the second human body, min_diss(P_1[face_index],P_2[all]) is the distance between the two keypoints with the smallest Euclidean distance between the face keypoint of the first human body and all human body keypoints of the second human body, and area_ratio is the balance coefficient.

[0110] Preferably, the balance coefficient is:

[0111]

[0112] Where area_ratio is the balance coefficient, box_area(P_1) is the area of ​​the smallest rectangle containing all the key points of the first human body, box_area(P_2) is the area of ​​the smallest rectangle containing all the key points of the second human body, img_area is the area of ​​the video image, min() is to take the minimum value, and max() is to take the maximum value.

[0113] The ratio is defined as the proportion between the smaller of the first and second human figures and the area of ​​the video image. The larger this ratio, the greater the likelihood that the first and second human figures are foreground rather than background, and correspondingly, the higher the limb distance score and face distance score of the first human figure relative to the second human figure; conversely, the smaller this ratio, the greater the likelihood that the first and second human figures are background rather than foreground, and correspondingly, the lower the limb distance score and face distance score of the first human figure relative to the second human figure.

[0114] The ratio is defined as the proportion between the smaller and larger human figures in the first and second human figures. A ratio close to 0 indicates that the first and second human figures represent an interaction between a foreground and background human figure, and correspondingly, the lower the limb distance score and face distance score of the first human figure relative to the second human figure. Conversely, a ratio close to 1 indicates that the first and second human figures occupy roughly the same area, and correspondingly, the higher the limb distance score and face distance score of the first human figure relative to the second human figure.

[0115] The balance coefficient indicates that if the area occupied by the first and second human bodies is roughly equal, and the larger the area occupied by the first and second human bodies, the larger the value of the balance coefficient, and the higher the limb distance score and face distance score of the first human body relative to the second human body.

[0116] S204: Determine the image score of the video image based on the limb distance scores and face distance scores corresponding to all human bodies contained in the video image.

[0117] In practice, the image score of a video image is determined based on the limb distance scores and face distance scores of all human bodies contained in the video image. This includes: finding the maximum value among the limb distance scores of all human bodies contained in the video image; finding the maximum value among the face distance scores of all human bodies contained in the video image; and using the sum of the two maximum values ​​as the image score of the video image.

[0118] Take, for example, a video frame from the previous text containing two human bodies, referred to as human body A and human body B.

[0119] In step S204, it is necessary to find the maximum value between the limb distance score of human body A relative to human body B and the limb distance score of human body B relative to human body A; find the maximum value between the face distance score of human body A relative to human body B and the face distance score of human body B relative to human body A; then, the sum of these two maximum values ​​is used as the image score of the video frame.

[0120] Take, for example, a video frame from the previous text containing three human bodies, referred to as human body A, human body B, and human body C.

[0121] In step S204, the maximum value needs to be found among the following limb distance scores: limb distance score of human body A relative to human body B, limb distance score of human body A relative to human body C, limb distance score of human body B relative to human body A, limb distance score of human body B relative to human body C, limb distance score of human body C relative to human body A, and limb distance score of human body C relative to human body B; the maximum value needs to be found among the following face distance scores: face distance score of human body A relative to human body B, face distance score of human body A relative to human body C, face distance score of human body B relative to human body A, face distance score of human body B relative to human body C, face distance score of human body C relative to human body A, and face distance score of human body C relative to human body B; then, the sum of these two maximum values ​​is used as the image score of the video frame.

[0122] The present application discloses a method for generating a dynamic cover, and correspondingly, the present application also discloses a device for generating a dynamic cover. The descriptions of the two in the specification can be referenced interchangeably.

[0123] Referring to Figure 3, which is a schematic diagram of the structure of a dynamic cover generation device disclosed in this application, the dynamic cover generation device includes a video preprocessing module 10, a key point detection module 20, an image score determination module 30, a target image set generation module 40, and a dynamic cover generation module 50.

[0124] in:

[0125] The video preprocessing module 10 is used to extract frames from the target video to obtain a video image set.

[0126] The key point detection module 20 is used to perform human key point detection on each frame of the video image set.

[0127] The image score determination module 30 is used to determine the image score of each video image in the video image set based on the key point detection results. Specifically, when the number of human figures in a video image exceeds a specified range, the image score is 0; when the number of human figures in a video image is within the specified range, the image score is positively correlated with the degree of interaction between the human figures in the video image.

[0128] The target image set generation module 40 is used to construct multiple candidate image sets based on the video image set, determine the comprehensive score of each candidate image set, and select a first number of target image sets from the multiple candidate image sets based on the principle of selecting the highest comprehensive score. The comprehensive score of each candidate image set is determined by the image scores of the video images contained in the candidate image set.

[0129] The dynamic cover generation module 50 is used to generate a dynamic cover for the target video based on a first set of target images.

[0130] The dynamic cover generation device disclosed in this application automatically generates dynamic covers for target videos without human intervention, which is highly efficient, takes very little time, and does not miss any exciting parts. In addition, the image score can intuitively reflect the degree of human interaction in the video images. Subsequently, the target image set is selected from multiple candidate image sets based on the principle of prioritizing the highest comprehensive score. This results in the selected target image set containing more human interaction scenes, thereby making the generated dynamic cover contain more human interaction scenes, which is conducive to attracting the user's attention.

[0131] Optionally, the image score determination module includes:

[0132] The human body number determination unit is used to determine the number of human bodies contained in a video image based on the key point detection results.

[0133] The first processing unit is used to determine the image score of the video image as a first value when the number of human bodies contained in the video image exceeds a specified range.

[0134] The second processing unit is used to determine the limb distance score and face distance score of each human body in the video image relative to each other human body, provided that the number of human bodies contained in the video image is within a specified range, and to determine the image score of the video image based on the limb distance scores and face distance scores corresponding to all human bodies contained in the video image.

[0135] Optionally, the second processing unit includes a limb distance score determination subunit, a face distance score determination subunit, and an image score determination subunit.

[0136] As one implementation method, the limb distance score determination subunit determines the limb distance score of the first human body relative to the second human body for any given video image, specifically as follows:

[0137] The distance scores of the limbs of the first human body relative to the second human body are determined according to the formula arm_dis_12=1 / min_diss(P_1[arm_index],P_2[all].

[0138] As another implementation, the limb distance score determination subunit determines the limb distance score of the first human body relative to the second human body for any given video image, specifically as follows:

[0139] The distance scores of the limbs of the first human body relative to the second human body are determined according to the formula arm_dis_12=area_ratio / min_diss(P_1[arm_index],P_2[all].

[0140] As one implementation method, the face distance score determination subunit determines the face distance score between a first human body and a second human body in any video image, specifically as follows:

[0141] The face distance score between the first human body and the second human body is determined according to the formula face_dis_12=1 / min_diss(P_1[face_index],P_2[all].

[0142] As another implementation, the face distance score determination subunit determines the face distance score between a first human body and a second human body in any video image, specifically as follows:

[0143] The face distance score between the first human body and the second human body is determined according to the formula face_dis_12=area_ratio / min_diss(P_1[face_index],P_2[all]).

[0144] Optional, the balance coefficient is:

[0145]

[0146] Where area_ratio is the balance coefficient, box_area(P_1) is the area of ​​the smallest rectangle containing all the key points of the first human body, box_area(P_2) is the area of ​​the smallest rectangle containing all the key points of the second human body, img_area is the area of ​​the video image, min() is to take the minimum value, and max() is to take the maximum value.

[0147] Optionally, the image score determination subunit determines the image score of the video image based on the limb distance scores and face distance scores corresponding to all human figures contained in the video image, specifically as follows:

[0148] Find the maximum limb distance score among all human figures contained in the video image; find the maximum face distance score among all human figures contained in the video image; and use the sum of the two maximum values ​​as the image score of the video image.

[0149] Optionally, the target image set generation module 40 constructs multiple candidate image sets based on the video image set. Specifically, it takes each frame of video image in the video image set as the starting point and obtains a second number of video images from the video image set to form multiple candidate image sets.

[0150] Optionally, the target image set generation module 40 selects at least a first number of target image sets from multiple candidate image sets based on the rule of high comprehensive score. Specifically, it selects a first number of target image sets from multiple candidate image sets based on the comprehensive score of all candidate image sets using the NMS algorithm.

[0151] This application also provides an electronic device.

[0152] Referring to Figure 4, which shows the hardware structure of an electronic device, the electronic device includes: a processor 401, a memory 402, a communication interface 403, and a communication bus 404.

[0153] In this embodiment, the processor 401, memory 402, and communication interface 403 communicate with each other via a communication bus 404. The communication bus 404 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0154] It should be noted that those skilled in the art will understand that the structure of the electronic device shown in Figure 4 does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in Figure 4, or combine certain components, or have different component arrangements.

[0155] The following section, with reference to Figure 4, provides a detailed description of each component of the electronic device.

[0156] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0157] The processor 401 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0158] The memory 402 may include memory such as high-speed random-access memory (RAM) and read-only memory (ROM), and may also include mass storage devices such as at least one disk storage device.

[0159] Communication interface 403 can be the interface of the communication module.

[0160] The memory 402 stores a program, and the processor 401 can call the program stored in the memory. The program is used for:

[0161] Frames are extracted from the target video to obtain a set of video images;

[0162] Human key point detection is performed on each frame of the video image set;

[0163] For each frame of video image in the video image set, an image score is determined based on the key point detection results of the video image. When the number of human bodies contained in the video image exceeds a specified range, the image score of the video image is a first value. When the number of human bodies contained in the video image is within the specified range, the image score of the video image is positively correlated with the degree of interaction of the human bodies in the video image.

[0164] Based on the video image set, multiple candidate image sets are constructed, and the comprehensive score of each candidate image set is determined. At least a first number of target image sets are selected from the multiple candidate image sets based on the principle of prioritizing those with higher comprehensive scores. The comprehensive score of a candidate image set is determined by the image scores of the video images contained in the candidate image set.

[0165] Based on the first set of target images, a dynamic cover is generated for the target video.

[0166] Optionally, the refined and extended functions of the program can be found in the description above.

[0167] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section.

[0169] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a dynamic cover, characterized in that, include: Frames are extracted from the target video to obtain a set of video images; Human key point detection is performed on each frame of the video image set; For each frame of the video image set, an image score is determined based on the keypoint detection results. When the number of human figures in the video image exceeds a specified range, the image score is a first value. When the number of human figures in the video image is within the specified range, for each human figure in the video image, the limb distance score and face distance score relative to each other human figure are determined. Based on the limb distance scores and face distance scores corresponding to all human figures in the video image, the image score of the video image is determined. Multiple candidate image sets are constructed based on the video image set, and the comprehensive score of each candidate image set is determined. At least a first number of target image sets are selected from the multiple candidate image sets based on the principle of prioritizing higher comprehensive scores. The comprehensive score of each candidate image set is determined by the image scores of the video images contained in the candidate image set. Based on the first number of target image sets, a dynamic cover image for the target video is generated.

2. The method according to claim 1, characterized in that, For any given video image, determine the limb distance score of the first human figure relative to the second human figure, including: according to the formula The limb distance scores of the first human body relative to the second human body are determined; for any video image, the facial distance scores between the first and second human bodies are determined, including: according to the formula The facial distance score between the first human body and the second human body is determined; where, The score is calculated based on the distance between the limbs of the first human body and the second human body. The score represents the facial distance between the first human body and the second human body. An index for key points of the limbs. Here are the coordinates of the key points of the limbs of the first human body. Index of facial landmarks Let be the coordinates of the facial landmarks of the first human body. These are the coordinates of all key points on the second human body. Let be the distance between the two key points that have the smallest Euclidean distance between the key points of the limbs of the first human body and all key points of the second human body. ) is the distance between the two key points with the smallest Euclidean distance between the facial key points of the first human body and all human body key points of the second human body.

3. The method according to claim 1, characterized in that, For any given video image, determine the limb distance score of the first human figure relative to the second human figure, including: according to the formula The limb distance scores of the first human body relative to the second human body are determined; for any video image, the facial distance scores between the first and second human bodies are determined, including: according to the formula The facial distance score between the first human body and the second human body is determined; where, The score is calculated based on the distance between the limbs of the first human body and the second human body. The score represents the facial distance between the first human body and the second human body. An index for key points of the limbs. Here are the coordinates of the key points of the limbs of the first human body. Index of facial landmarks Let be the coordinates of the facial landmarks of the first human body. These are the coordinates of all key points on the second human body. Let be the distance between the two key points that have the smallest Euclidean distance between the key points of the limbs of the first human body and all key points of the second human body. The distance is the distance between the two key points that have the smallest Euclidean distance between the facial key points of the first human body and all key points of the second human body. This is the balance coefficient.

4. The method according to claim 3, characterized in that, The balance coefficient is: in, For balance coefficient, Let the area be the smallest rectangle that contains all the key points of the first human body. The area of ​​the smallest rectangle containing all the key points of the second human body is given by , where img_area is the area of ​​the video image, min() is used to find the minimum value, and max() is used to find the maximum value.

5. The method according to claim 1, characterized in that, The step of determining the image score of the video image based on the limb distance scores and face distance scores of all human bodies contained in the video image includes: finding the maximum value among the limb distance scores of all human bodies contained in the video image; finding the maximum value among the face distance scores of all human bodies contained in the video image; and using the sum of the two maximum values ​​as the image score of the video image.

6. The method according to claim 1, characterized in that, The step of constructing multiple candidate image sets based on the video image set includes: taking each frame of video image in the video image set as a starting point, obtaining a second number of video images from the video image set to form multiple candidate image sets.

7. The method according to claim 1, characterized in that, The step of selecting a first number of target image sets from the multiple candidate image sets based on the principle of prioritizing those with higher overall scores includes: selecting a first number of target image sets from the multiple candidate image sets using the NMS algorithm based on the overall scores of all candidate image sets.

8. A dynamic cover generation device, characterized in that, include: The video preprocessing module is used to extract frames from the target video to obtain a video image set; The key point detection module is used to perform human key point detection on each frame of the video image set. The image score determination module is used to determine the image score of each video image in the video image set based on the key point detection results of the video image. When the number of human figures in the video image exceeds a specified range, the image score of the video image is 0. When the number of human figures in the video image is within the specified range, for each human figure in the video image, the limb distance score and face distance score of that human figure relative to each other human figure are determined. Based on the limb distance scores and face distance scores corresponding to all human figures in the video image, the image score of the video image is determined. The target image set generation module is used to construct multiple candidate image sets based on the video image set, determine the comprehensive score of each candidate image set, and select a first number of target image sets from the multiple candidate image sets based on the principle of prioritizing those with higher comprehensive scores. The comprehensive score of the candidate image set is determined by the image scores of the video images contained in the candidate image set. The dynamic cover generation module is used to generate a dynamic cover for the target video based on the first number of target image sets.

9. An electronic device, characterized in that, It includes a processor and a memory; the memory is used to store a program; the processor is used to execute the program to implement the various steps of the method as described in any one of claims 1 to 7.

Images