A short video evaluation and creation assistance method and system based on visual understanding

Abstract

The application discloses a short video evaluation and creation assistance method and system based on visual understanding, the method comprising: acquiring short video data, extracting key frames from the video; detecting the key frames to obtain the proportion information of the face region position in the picture; identifying the expression category of the detected face region, and counting the number of expression category changes in adjacent time windows; estimating the global motion amount of the picture based on the optical flow information between adjacent frames, calculating the lens shake degree and forming the picture stability index; obtaining the video frame change rate by calculating the pixel difference and color histogram difference between adjacent frames; and fusing the above indexes to output a comprehensive score of the video hit potential. The application analyzes visual features such as the state of the character in the video picture, the expression change, the picture stability and the rhythm change, constructs an interpretable hit potential evaluation model, and provides objective, real-time and executable intelligent assistance decision for short video and live content creation.

Description

Technical Field

[0001] This application relates to the fields of computer vision and artificial intelligence application technology, and in particular to a method and system for short video evaluation and creation assistance based on visual understanding. Background Technology

[0002] Current short video content production processes heavily rely on human experience and lack objective, quantitative assessment methods for evaluating the appeal and dissemination potential of video content. Existing technologies for short video content analysis mainly include the following categories:

[0003] Content feedback mechanisms based on platform recommendation algorithms: This type of technology only analyzes the dissemination effect after the video is published. Creators cannot obtain guidance during the content production stage, and the platform algorithm is opaque and not reusable.

[0004] Video editing and automatic generation tools, such as automatic editing and high-energy clip extraction tools, mainly improve production efficiency, but do not systematically evaluate the appeal of the content and its potential to become a hit.

[0005] Video Quality Assessment (VQA): This type of technology mainly evaluates technical quality such as video clarity and compression distortion, but it cannot reflect the content's expressiveness and the user's viewing experience.

[0006] Large-model-based text or strategy analysis tools: These solutions focus on text generation or creative assistance, lack a deep visual understanding of the video footage itself, and have insufficient interpretability.

[0007] In summary, current technologies cannot yet achieve quantitative evaluation and creative assistance for short video content based on visual semantics during the creation stage. The main problems are as follows:

[0008] The assessment of short video "viral potential" relies on subjective human judgment and is difficult to standardize;

[0009] Existing video analytics tools mostly focus on editing efficiency or recommendation result analysis, and cannot provide decision support during the creation stage;

[0010] There is a lack of interpretable analysis of the video's visual expressiveness (such as the characters' states, rhythm changes, and image stability).

[0011] Existing systems rely heavily on cloud processing, making it difficult to meet the requirements for real-time performance, privacy, and low-cost deployment.

[0012] Therefore, there is an urgent need for a technical solution that can intelligently evaluate short videos and output actionable creative suggestions in the pre-production, production, and post-production stages of content creation. Summary of the Invention

[0013] This application provides a short video evaluation and creation assistance method and system based on visual understanding. Its advantage is that by analyzing visual features such as the state of people, changes in facial expressions, stability of the scene and changes in rhythm in the video, an interpretable potential hit evaluation model is constructed, providing objective, real-time and executable intelligent auxiliary decision-making for the creation of short video and live streaming content.

[0014] The technical solution of this application is as follows:

[0015] On the one hand, this application provides a short video evaluation and creation assistance method based on visual understanding, including the following steps:

[0016] S1: Obtain the short video data to be analyzed, extract key frames from the video according to the preset frame rate, and generate an image sequence with timestamps;

[0017] S2: Detect the acquired keyframes to obtain the location of the face region and its proportion in the image;

[0018] S3: Recognize the expression category of the detected face region and count the number of expression category changes within adjacent time windows to obtain the expression change rate index;

[0019] S4: Estimate the global motion of the image based on the optical flow information between adjacent frames, calculate the degree of camera shake, and form an image stability index;

[0020] S5: Obtain the video frame change rate by calculating the pixel difference and color histogram difference between adjacent frames;

[0021] S6: Input facial proportion information, expression change rate index, image stability index and frame change rate into the hit potential assessment model, and output a comprehensive score of video hit potential.

[0022] Furthermore, step S1 includes the following steps: reading the video's frame rate (FPS); calculating the target frame extraction sequence according to a preset frame extraction interval; extracting corresponding frames in chronological order and assigning a timestamp to each frame, resulting in the timestamped image sequence:

[0023] .

[0024] Furthermore, in step S2, the face region is detected using the YOLOv5-face lightweight object detection model to obtain its location and size information; the formula for calculating the face area ratio is:

[0025]

[0026] The width of the face frame;

[0027] The height of the human face frame;

[0028] Image width;

[0029] This represents the image height.

[0030] Furthermore, step S3 includes the following steps: normalizing the face region; inputting it into the expression classification model to obtain expression category labels; and counting the number of times the expression category changes within adjacent time windows.

[0031] The rate of change of facial expressions is obtained by the following formula:

[0032]

[0033] R represents the rate of change in facial expression;

[0034] N represents the number of times the expression category changes;

[0035] T represents the duration of time.

[0036] Furthermore, step S4 includes the following steps: calculating optical flow information between adjacent frames; estimating the global motion of the entire image; removing local motion in the face region; and using the remaining motion to characterize the degree of lens shake.

[0037] Image stability is obtained using the following formula:

[0038]

[0039] M is the average global optical flow amplitude.

[0040] S represents stability, with a value range of 0-1.

[0041] Furthermore, step S5 includes the following steps: calculating the pixel difference and color histogram difference between adjacent frames; obtaining the time series of change intensity; and smoothing the change intensity.

[0042] The frame rate of change is obtained by the following formula:

[0043]

[0044] FCR represents the frame rate of change, F_t represents the current frame, and F_t-1 represents the previous frame.

[0045] Furthermore, in step S6, the potential hit product assessment model involves weighted fusion based on preset weights to generate a comprehensive score.

[0046]

[0047] Score = Overall score;

[0048] =Face change rate;

[0049] =Frame change rate;

[0050] =Face proportion;

[0051] S = stability;

[0052] w1, w2, w3, w4 = weighting coefficients, satisfying w1 + w2 + w3 + w4 = 1.

[0053] Furthermore, step S7 is included: generating natural language creative suggestions for video rhythm, composition, and character performance based on various visual indicators and preset rules.

[0054] On another front, this application provides a short video evaluation and creation assistance system based on visual understanding, comprising:

[0055] The keyframe extraction module is used to acquire the short video data to be analyzed, extract keyframes from the video according to a preset frame rate, and generate a time-stamped image sequence.

[0056] The face and subject detection module is used to detect the acquired keyframes and obtain information on the location of face regions and their proportion in the image.

[0057] The facial expression change analysis module is used to identify the facial expression category of the detected face region and count the number of facial expression category changes within adjacent time windows to obtain the facial expression change rate index.

[0058] The stability analysis module is used to estimate the global motion of the image based on the optical flow information between adjacent frames, calculate the degree of camera shake, and form an image stability index.

[0059] The rhythm analysis module is used to obtain the video frame change rate by calculating the pixel differences and color histogram differences between adjacent frames;

[0060] It also includes a hit potential assessment module, which inputs facial proportion information, expression change rate index, image stability index and frame change rate into the hit potential assessment model, and outputs a comprehensive score of the video's hit potential.

[0061] Furthermore, it also includes a creative assistance suggestion module, which generates natural language creative suggestions for video rhythm, composition, and character performance based on various visual indicators and preset rules.

[0062] In summary, the beneficial effects of this application are as follows:

[0063] Transform short video creation experience into calculable and interpretable visual metrics;

[0064] To achieve intelligent evaluation of short video content during the creation stage, rather than post-creation analysis;

[0065] Provide specific and actionable suggestions for content creation optimization to improve content production efficiency;

[0066] The system can be deployed on edge computing devices to achieve real-time analysis and privacy protection;

[0067] It is suitable for various content production scenarios such as short videos, live streaming, and e-commerce. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the structure of the short video evaluation and creation assistance system based on visual understanding, as described in an embodiment of this application.

[0069] Figure 2 This is a schematic diagram of the time distribution of facial expression changes in an embodiment of this application. Detailed Implementation

[0070] The specific embodiments of this application are described in detail below with reference to the accompanying drawings.

[0071] A specific embodiment of this application provides a short video evaluation and creation assistance method based on visual understanding, including the following steps:

[0072] S1: Video input and frame extraction;

[0073] Acquire short video data to be analyzed, extract keyframes from the video according to a preset frame rate, and generate a timestamped image sequence;

[0074] This step takes a short video file (MP4 / AVI / MOV) as input, with a duration of 5–60 seconds, and reads the video's frame rate (FPS). Based on a preset frame extraction interval, it calculates the target frame extraction sequence; extracts corresponding frames in chronological order, assigning a timestamp to each frame. The frame extraction frequency is lower than the original video frame rate to reduce computational burden; and retains the timestamp information for subsequent rhythm analysis and high-energy segment localization. The output is a time-stamped image sequence:

[0075] .

[0076] S2: Face and subject detection;

[0077] The acquired keyframes are inspected to obtain the location of the face region and its proportion in the image.

[0078] This step takes a sequence of keyframe images as input. These keyframes are then fed into a pre-trained face detection model, specifically the YOLOv5-face lightweight object detection model. Each frame outputs the coordinates of the face bounding box. If multiple faces are detected, the one with the largest area is selected as the primary face. The YOLOv5-face lightweight object detection model detects the face region without performing identity verification, only acquiring location and size information. The formula for calculating the face area ratio is:

[0079]

[0080] The width of the face frame;

[0081] The height of the human face frame;

[0082] Image width;

[0083] This represents the image height.

[0084] Output the sequence of face positions and face proportions for each frame.

[0085] S3: Calculation of the rate of change in human facial expressions;

[0086] The detected facial regions are used to identify the expression category, and the number of times the expression category changes within adjacent time windows is counted to obtain the expression change rate index.

[0087] This step takes a sequence of face region images as input, normalizes the face regions, inputs them into an expression classification model to obtain expression category labels, and counts the number of times the expression category changes within adjacent time windows.

[0088] The rate of change of facial expressions is obtained by the following formula:

[0089]

[0090] R represents the rate of change in facial expression;

[0091] N represents the number of times the expression category changes, such as Figure 2 As shown;

[0092] T represents the duration of time.

[0093] In this embodiment, the number of expression categories can be set to 5–7; the focus is on "variation" rather than semantic precision. Output of this step:

[0094] 1. Facial expression change rate value;

[0095] 2. Time distribution of facial expression changes.

[0096] S4: Image stability analysis;

[0097] The global motion of the image is estimated based on the optical flow information between adjacent frames, the degree of camera shake is calculated, and an image stability index is formed.

[0098] This step takes adjacent keyframe images as input. It calculates optical flow information between adjacent frames; estimates the global motion of the entire image; removes local motion from the face region; and uses the remaining motion to characterize the degree of camera shake.

[0099] Image stability is obtained using the following formula:

[0100]

[0101] M is the average global optical flow amplitude.

[0102] S represents stability, with a value range of 0-1.

[0103] S5: Frame Change Rate and Rhythm Analysis;

[0104] The video frame change rate is obtained by calculating the pixel differences and color histogram differences between adjacent frames;

[0105] This step takes a keyframe sequence as input, calculates the pixel difference and color histogram difference between adjacent frames, obtains a time series of change intensity, smooths the change intensity, and shows that the frame change rate reflects scene switching and motion density; the change peak corresponds to high-energy segments.

[0106] The frame rate of change is obtained by the following formula:

[0107]

[0108] FCR represents the frame rate of change, F_t represents the current frame, and F_t-1 represents the previous frame.

[0109] This step outputs:

[0110] 1. Video rhythm change curve;

[0111] 2. High-energy fragment time interval.

[0112] S6: Input facial proportion information, expression change rate index, image stability index and frame change rate into the hit potential assessment model, and output a comprehensive score of video hit potential.

[0113] This step takes the following inputs: face proportion, expression change rate, stability, and frame change rate; each visual feature indicator is normalized, and the potential hit assessment model uses weighted fusion based on preset weights; a comprehensive score is generated.

[0114]

[0115] Score = Overall score;

[0116] =Face change rate;

[0117] =Frame change rate;

[0118] =Face proportion;

[0119] S = stability;

[0120] w1, w2, w3, w4 = weighting coefficients, satisfying w1 + w2 + w3 + w4 = 1.

[0121] Output a potential hit rating (0–100).

[0122] Other embodiments of this application also include S7: Generating creative assistance suggestions;

[0123] Based on various visual metrics and preset rules, natural language creative suggestions are generated for video rhythm, composition, and character performance.

[0124] This step takes as input various visual indicators and scoring results, compares the indicators with preset thresholds, matches the corresponding creative rule templates, and generates natural language suggestions.

[0125] Output example:

[0126] "The pace of the beginning of the video is too slow; it is recommended to compress the first 3 seconds."

[0127] "The characters' facial expressions are not varied enough; we suggest enhancing their interactive performance."

[0128] Another specific embodiment of this application provides a short video evaluation and creation assistance system based on visual understanding, including:

[0129] The keyframe extraction module is used to acquire the short video data to be analyzed, extract keyframes from the video according to a preset frame rate, and generate a time-stamped image sequence.

[0130] The face and subject detection module is used to detect the acquired keyframes and obtain information on the location of face regions and their proportion in the image.

[0131] The facial expression change analysis module is used to identify the facial expression category of the detected face region and count the number of facial expression category changes within adjacent time windows to obtain the facial expression change rate index.

[0132] The stability analysis module is used to estimate the global motion of the image based on the optical flow information between adjacent frames, calculate the degree of camera shake, and form an image stability index.

[0133] The rhythm analysis module is used to obtain the video frame change rate by calculating the pixel differences and color histogram differences between adjacent frames;

[0134] It also includes a hit potential assessment module, which inputs facial proportion information, expression change rate index, image stability index and frame change rate into the hit potential assessment model, and outputs a comprehensive score of the video's hit potential.

[0135] The creative assistance suggestion module is used to generate natural language creative suggestions for video rhythm, screen composition and character performance based on various visual indicators and preset rules.

[0136] The visual understanding-based short video evaluation and creation assistance system evaluates short videos according to the steps in the visual understanding-based short video evaluation and creation assistance method described above.

[0137] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of this application, and these all fall within the protection scope of this application.

Claims

1. A short video evaluation and creation assistance method based on visual understanding, characterized in that, Includes the following steps: S1: Obtain the short video data to be analyzed, extract key frames from the video according to the preset frame rate, and generate an image sequence with timestamps; S2: Detect the acquired keyframes to obtain the location of the face region and its proportion in the image; S3: Recognize the expression category of the detected face region and count the number of expression category changes within adjacent time windows to obtain the expression change rate index; S4: Estimate the global motion of the image based on the optical flow information between adjacent frames, calculate the degree of camera shake, and form an image stability index; S5: Obtain the video frame change rate by calculating the pixel difference and color histogram difference between adjacent frames; S6: Input facial proportion information, expression change rate index, image stability index and frame change rate into the hit potential assessment model, and output a comprehensive score of video hit potential.

2. The short video evaluation and creation assistance method based on visual understanding according to claim 1, characterized in that, Step S1 includes the following steps: reading the video frame rate (FPS); calculating the target frame extraction sequence according to the preset frame extraction interval; extracting the corresponding frames in chronological order and assigning a timestamp to each frame; the timestamped image sequence is as follows: 。 3. The short video evaluation and creation assistance method based on visual understanding according to claim 2, characterized in that, In step S2, the face region is detected using the YOLOv5-face lightweight object detection model to obtain its location and size information; the formula for calculating the face area ratio is: The width of the face frame; The height of the human face frame; Image width; This represents the image height.

4. The short video evaluation and creation assistance method based on visual understanding according to claim 3, characterized in that, Step S3 includes the following steps: normalizing the face region; inputting it into the expression classification model to obtain expression category labels; and counting the number of times the expression category changes within adjacent time windows. The rate of change of facial expressions is obtained by the following formula: R represents the rate of change in facial expression; N represents the number of times the expression category changes; T represents the duration of time.

5. The short video evaluation and creation assistance method based on visual understanding according to claim 4, characterized in that, Step S4 includes the following steps: calculating optical flow information between adjacent frames; estimating the global motion of the entire image; removing local motion in the face region; and using the remaining motion to characterize the degree of lens shake. Image stability is obtained using the following formula: M is the average global optical flow amplitude. S represents stability, with a value range of 0-1.

6. The short video evaluation and creation assistance method based on visual understanding according to claim 5, characterized in that, Step S5 includes the following steps: calculating the pixel difference and color histogram difference between adjacent frames; obtaining the time series of intensity changes; and smoothing the intensity changes. The frame rate of change is obtained by the following formula: FCR represents the frame rate of change, F_t represents the current frame, and F_t-1 represents the previous frame.

7. The short video evaluation and creation assistance method based on visual understanding according to claim 1, characterized in that, In step S6, the potential hit product assessment model uses a weighted fusion method based on preset weights to generate a comprehensive score. Score = Overall score; =Face change rate; =Frame change rate; =Face proportion; S = stability; w1, w2, w3, w4 = weighting coefficients, satisfying w1 + w2 + w3 + w4 = 1.

8. The short video evaluation and creation assistance method based on visual understanding according to claim 6, characterized in that, It also includes step S7: generating natural language creation suggestions for video rhythm, composition and character performance based on various visual indicators and preset rules.

9. A short video evaluation and creation assistance system based on visual understanding, characterized in that, include: The keyframe extraction module is used to acquire the short video data to be analyzed, extract keyframes from the video according to a preset frame rate, and generate a time-stamped image sequence. The face and subject detection module is used to detect the acquired keyframes and obtain information on the location of face regions and their proportion in the image. The facial expression change analysis module is used to identify the facial expression category of the detected face region and count the number of facial expression category changes within adjacent time windows to obtain the facial expression change rate index. The stability analysis module is used to estimate the global motion of the image based on the optical flow information between adjacent frames, calculate the degree of camera shake, and form an image stability index. The rhythm analysis module is used to obtain the video frame change rate by calculating the pixel differences and color histogram differences between adjacent frames; It also includes a hit potential assessment module, which inputs facial proportion information, expression change rate index, image stability index and frame change rate into the hit potential assessment model, and outputs a comprehensive score of the video's hit potential.

10. The short video evaluation and creation assistance system based on visual understanding according to claim 9, characterized in that, It also includes a creative assistance suggestion module, which generates natural language creative suggestions for video rhythm, composition and character performance based on various visual indicators and preset rules.

Images