A method for automatically scoring and optimizing shot quality of sketch material

Abstract

The present application relates to the technical field of multimedia data processing, and more particularly to a short drama material shot quality automatic scoring and optimization method, comprising the following steps: S1, acquiring an original short drama video stream and associated screenplay data, and using boundary detection logic to segment the original short drama video stream into independent candidate shot segments; S2, performing image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing picture physical attributes and character performance information, and simultaneously analyzing the screenplay data to extract a text feature vector containing scene emotion and key action description. In the present application, a multimodal joint embedding space is constructed to calculate the semantic similarity of vision and text and generate a consistency weight factor accordingly, thereby combining objective image technical indicators with subjective script content description, and effectively solving the problem that traditional scoring methods are difficult to eliminate clear picture quality but deviate from the script material.

Description

Technical Field

[0001] This invention relates to the field of multimedia data processing technology, and in particular to a method for automatically scoring and optimizing the shot quality of short drama footage. Background Technology

[0002] With the explosive growth of the short video industry, short dramas are loved by audiences for their short production cycle and tight plot. In the actual shooting and production process, in order to capture the best performance and scene atmosphere, the production team often shoots the same storyboard multiple times, resulting in a massive amount of raw video material. How to efficiently select usable shots that meet both technical standards and narrative requirements from this massive amount of material has become a key link in improving post-production efficiency.

[0003] Most existing video material selection technologies rely on traditional no-reference image quality assessment algorithms, which focus on detecting objective technical indicators such as image sharpness, noise, exposure, and color. However, this single-dimensional assessment method cannot understand the semantic relationship between video content and storyboard script. As a result, invalid materials that are clear in image quality but have incorrect character performances, actions that deviate from the script description, or are inconsistent with the plot atmosphere cannot be effectively identified and eliminated. Ultimately, a lot of manual effort is still required for secondary review and screening. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides an automatic scoring and selection method for the shot quality of short drama materials, aiming to improve the problem that existing video screening technologies are mostly limited to objective quality assessment of underlying visual features such as image clarity and noise, and have difficulty understanding the semantic relationship between the image content and the storyboard.

[0005] This invention provides the following technical solution: an automatic scoring and optimization method for shot quality of short drama footage, comprising the following steps:

[0006] S1. Obtain the original short drama video stream and associated storyboard data, and use boundary detection logic to segment the original short drama video stream into independent candidate shot segments;

[0007] S2. Perform image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing the physical attributes of the scene and the performance information of the characters. At the same time, parse the storyboard data to extract a text feature vector containing the scene's emotions and key action descriptions.

[0008] S3. Calculate the semantic similarity value between the visual feature vector and the text feature vector, and normalize the semantic similarity value to generate a consistency weight factor.

[0009] S4. Input the visual feature vector into the preset quality assessment model to output the basic quality score, and use the consistency weight factor to perform weighted calculation on the basic quality score to obtain the comprehensive quality score of each candidate shot segment.

[0010] S5. Calculate the feature distance between segments based on the visual feature vector, and cluster multiple candidate shot segments whose feature distance meets a preset threshold into the same scene set;

[0011] S6. Traverse each set of sessions, sort the elements in the set according to the comprehensive quality score, extract the segment with the highest score as the preferred material and output it.

[0012] Preferably, in step S1, the original short drama video stream is segmented into several independent candidate shot segments using boundary detection logic, specifically including:

[0013] The original short drama video stream is processed by frame extraction at preset time intervals to obtain a continuous sequence of frame images, and each frame image in the sequence is converted from RGB color space to a color space with separate luminance and chrominance.

[0014] The color distribution difference values ​​of two adjacent frames in the separated color space are calculated respectively to form an inter-frame difference sequence that varies with time.

[0015] The local dynamic threshold of the inter-frame difference sequence is calculated using sliding window logic. The local dynamic threshold is composed of the sum of the mean and standard deviation of the data within the window.

[0016] The elements in the inter-frame difference sequence are compared with the local dynamic threshold. When the difference value of a specific frame is greater than the local dynamic threshold and the difference value is a local maximum within a preset neighborhood range, the video stream is truncated into the candidate shot segments based on the time position of the frame.

[0017] Preferably, in step S2, image feature extraction processing is performed on each candidate shot segment, specifically including:

[0018] Keyframes are extracted from the candidate shot segments, and shallow texture features and deep semantic features of the keyframes are extracted through a neural network model and combined to generate global physical property features.

[0019] Identify the facial regions of the person in the keyframe, construct the region of interest of the face, and extract the facial motion unit parameters and emotion probability distribution of the region to generate facial emotion features;

[0020] The temporal frame sequence of the candidate shot fragment is input into the spatiotemporal feature extraction network to extract spatiotemporal action features;

[0021] The visual feature vector is obtained by multimodal splicing and dimensional mapping of the global physical attribute features, the facial emotion features, and the spatiotemporal action features.

[0022] Preferably, in step S3, calculating the semantic similarity value between the visual feature vector and the text feature vector specifically includes:

[0023] A multimodal joint embedding space is constructed, and the visual feature vector and the text feature vector are mapped to the joint embedding space of the same dimension using two independent projection transformation matrices respectively.

[0024] Calculate the spatial distance or cosine of the angle between the mapped visual vector and the text vector to obtain the original matching score;

[0025] The original matching score is normalized and mapped to obtain the semantic similarity value;

[0026] The consistency weight factor is generated based on the semantic similarity value, wherein when the semantic similarity value is lower than a preset penalty threshold, the consistency weight factor is set to a value less than 1.

[0027] Preferably, in step S4, the weighted calculation of the basic quality score using the consistency weighting factor specifically includes:

[0028] The visual feature vector is input into the quality assessment network to predict the technical index score and aesthetic composition score of the image. The weighted sum is then used to output the basic quality score.

[0029] Set a basic quality threshold. If the basic quality score is lower than the basic quality threshold, then set the overall quality score of the candidate shot segment to zero.

[0030] If the basic quality score meets the basic quality threshold, the basic quality score is calculated by multiplication using the consistency weighting factor. When the consistency weighting factor is greater than the benchmark value, the basic quality score is increased; when the consistency weighting factor is less than the benchmark value, the basic quality score is decreased.

[0031] The calculated overall quality score is then smoothed in the temporal domain to obtain the final overall quality score assigned to the candidate shot segment.

[0032] Preferably, in step S5, calculating the feature distance between segments based on the visual feature vector specifically includes:

[0033] The visual feature vectors of the candidate shot segments to be compared are subjected to channel weighting processing. The feature channel representing the background of the scene is assigned a first weight, and the feature channel representing local action changes is assigned a second weight. The value of the first weight is greater than the value of the second weight.

[0034] Calculate the distance between the feature vectors after channel weighting, and use it as the basic visual difference value;

[0035] The timestamp information of the candidate shot segments is obtained and the time interval is calculated. The time interval is then converted into a time penalty term using a preset mapping relationship.

[0036] The feature distance is obtained by weighted summation of the basic visual difference value and the time penalty term.

[0037] Preferably, in step S5, clustering multiple candidate shot segments whose feature distance satisfies a preset threshold into the same scene set specifically includes:

[0038] Initialize the list of scene sets, create the first scene set, and use the first candidate shot as the initial cluster center of the set;

[0039] Traverse subsequent candidate shot segments in chronological order and calculate the feature distance between the current segment and the cluster center of each existing set of scenes.

[0040] The minimum value among the feature distances is selected. If the minimum value is less than the preset scene determination threshold, the current segment is assigned to the corresponding scene set, and the cluster center of the set is updated using the visual feature vector of the current segment.

[0041] If the minimum value does not meet the scene determination threshold, a new scene set is created and the current segment is set as the new cluster center;

[0042] Repeat the above steps until all candidate shot segments have been traversed.

[0043] Preferably, in step S6, the elements within the set are sorted according to the comprehensive quality score, specifically including:

[0044] All candidate shot segments within the set of scenes are sorted in descending order according to the comprehensive quality score to generate an initial candidate queue.

[0045] Move the top-ranked candidate shot in the initial candidate queue into the preferred result list;

[0046] Traverse the remaining candidate shot segments in the initial candidate queue and calculate the content repetition between the current segment and the existing material in the preferred result list.

[0047] The repetition rate of the image content is compared with a preset repetition threshold. If it is higher than the repetition threshold, the current segment is determined to be a repetitive segment and is removed or marked.

[0048] If the repetition threshold is lower than the specified threshold, the current segment is retained.

[0049] The materials in the preferred result list are filtered by duration, and segments with a duration less than the preset minimum usable duration are removed, and the final preferred materials are output.

[0050] The present invention has the following beneficial effects:

[0051] 1. In this invention, by constructing a multimodal joint embedding space to calculate the semantic similarity between vision and text and generating a consistency weight factor accordingly, objective image technical indicators are combined with subjective script content descriptions, effectively solving the problem that traditional scoring methods are unable to eliminate images with clear quality but content that deviates from the script material.

[0052] 2. In this invention, a feature channel differential weighting strategy is adopted and a time interval is introduced as a penalty term. While weakening the interference of human movement, the ability to recognize scene background is enhanced, and the shots that are not continuous in time but are shot in the same scene are automatically clustered into independent scene sets.

[0053] 3. In this invention, structural similarity index is used to detect the repetition of screen content in high-scoring materials in the preferred queue. Under the premise of ensuring that the image quality of the materials meets the requirements, redundant shots under the same camera position are eliminated, realizing the automated initial screening of short drama materials from raw data to editable sequences. Attached Figure Description

[0054] Figure 1 This is a flowchart of an automatic scoring and optimization method for shot quality of short drama material proposed in this invention. Detailed Implementation

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] This invention provides an automatic scoring and optimization method for shot quality of short drama footage, such as... Figure 1 As shown, it includes the following steps:

[0057] S1. Obtain the original short drama video stream and associated storyboard data, and use boundary detection logic to segment the original short drama video stream into independent candidate shot segments.

[0058] Furthermore, in S1, boundary detection logic is used to segment the original short drama video stream into several independent candidate shot segments, specifically including:

[0059] The original short drama video stream is processed by frame extraction at preset time intervals to obtain a continuous sequence of frame images. Each frame in the sequence is then converted from the RGB color space to a color space that separates luminance and chrominance.

[0060] The color distribution difference between two adjacent frames in the separated color space is calculated to form an inter-frame difference sequence that varies over time.

[0061] The local dynamic threshold of the inter-frame difference sequence is calculated using sliding window logic. The local dynamic threshold is composed of the sum of the mean and standard deviation of the data within the window.

[0062] The elements in the inter-frame difference sequence are compared with the local dynamic threshold. When the difference value of a specific frame is greater than the local dynamic threshold and the difference value is a local maximum within a preset neighborhood, the video stream is truncated into candidate shot segments based on the time position of that frame.

[0063] Specifically, first, frame extraction and color space conversion of the video stream are performed, defining the input original short drama video stream as... According to the preset time interval (For example, every 0.5 seconds) Discretization sampling is performed to obtain a temporally continuous sequence of image frames. ,in The total number of frames is [number]. To reduce the interference of overall fluctuations in illumination intensity on content detection, each frame in the sequence is [number]. The process involves converting the original RGB color space to a color space that separates luminance and chrominance, such as YUV or CIELAB. Taking the conversion to YUV as an example, let's assume... , is the coordinate in the image The pixel value at that location, the converted luminance component and chromaticity components Calculated using the linear transformation matrix, the feature representation of each frame image in the separated color space is as follows: This feature preserves the structural information of the image and decouples brightness from color.

[0064] Next, the color distribution difference between adjacent frames is calculated, for each adjacent frame in the sequence. and The pixel value differences in the luminance and chroma channels are calculated separately to generate a quantized difference value. The difference in numerical value It is represented by the sum of the absolute values ​​of the pixel differences between each channel, and its calculation formula is as follows:

[0065] ;

[0066] in, and These are the height and width of the image, respectively. These are the weighting coefficients for the corresponding channels, typically the luminance channel. The weight is set higher than that of the chroma channel. Representing the individual channel components in the separate color space, an inter-frame difference sequence is formed by traversing and calculating the entire sequence, thus generating a time-varying sequence. This sequence reflects the degree of dynamic change in the video content.

[0067] Subsequently, a local dynamic threshold is constructed to adapt to the rhythmic changes of the video content, using a threshold of length [missing information]. The sliding window in the inter-frame difference sequence Swipe up to view the difference value at the current time. Its corresponding local dynamic threshold Based on the statistical characteristics of the data within the window, the set of differences within the window is calculated first. mean and standard deviation The calculation formula is as follows:

[0068] ;

[0069] Based on the above statistics, a local dynamic threshold is defined. A linear combination of the mean and standard deviation:

[0070] ;

[0071] in, This is the sensitivity adjustment coefficient, usually set to 1. This threshold setting method can automatically adjust the judgment standard according to the intensity of the current video segment, automatically increasing the threshold in intense fighting scenes and automatically decreasing the threshold in static dialogue scenes.

[0072] Finally, boundary determination and video stream truncation are performed, separating each element in the inter-frame difference sequence. Its corresponding local dynamic threshold To determine whether a frame position is a shot transition boundary through a comparison, two conditions must be met simultaneously: firstly, the current difference value... The value must be strictly greater than the local dynamic threshold. ,Right now Secondly, the difference value It must be a local maximum within a preset neighborhood, that is, it must satisfy... and When all of the above conditions are met, the first... Frame and the There are shot transitions between frames; these points in time are recorded as breakpoints. The original short drama video stream is then reconstructed using all identified breakpoints. Physically truncate the timeline to output a series of independent candidate shot fragments. .

[0073] This effectively overcomes the shortcomings of a single fixed threshold that cannot take into account the fast and slow rhythm switching of short dramas. It reduces false detections caused by lighting flicker by using luminance-color separation, and suppresses interference caused by rapid object movement by using local dynamic thresholds and local maxima, thereby achieving high-precision automated shot segmentation of short drama materials.

[0074] S2. Perform image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing the physical attributes of the image and the performance information of the characters. At the same time, parse the storyboard data to extract a text feature vector containing the scene's emotions and key action descriptions.

[0075] Furthermore, in S2, image feature extraction processing is performed on each candidate shot segment, specifically including:

[0076] Keyframes are extracted from candidate shot segments. Shallow texture features and deep semantic features of the keyframes are extracted through a neural network model and combined to generate global physical property features.

[0077] Identify the facial regions of people in keyframes, construct the region of interest for the face, and extract the facial motion unit parameters and emotion probability distribution of the region to generate facial emotion features.

[0078] The temporal frame sequence of the candidate shot fragments is input into the spatiotemporal feature extraction network to extract spatiotemporal action features;

[0079] The global physical attribute features, facial emotion features, and spatiotemporal action features are multimodal spliced ​​and dimensionally mapped to obtain the visual feature vector.

[0080] Specifically, firstly, text feature extraction is performed on the storyboard data to obtain storyboard data associated with the current candidate shot segment in terms of time or scene number. This data includes adjective descriptions of the scene atmosphere, descriptions of characters' psychological activities, and text descriptions of key physical actions. The above text content is preprocessed by word segmentation and stop word removal to obtain a normalized word sequence. This word sequence is then input into a pre-trained converter bidirectional encoding representation model, namely the BERT model. The model's multi-layer self-attention mechanism captures the contextual semantic relationships in the text. The vectors corresponding to the classification labels representing the overall semantics of the sentence are extracted from the model's output layer and defined as text feature vectors. This vector numerically represents the expected emotional tone and key action requirements of the scene in the script for the current shot.

[0081] Secondly, keyframe extraction and global physical property feature construction of candidate shot segments are performed. The duration information of the candidate shot segments is read, and the frame image located at the middle moment of the segment is selected as the keyframe representing the segment. The keyframe is input into a deep convolutional neural network model, such as a residual network pre-trained on the ImageNet dataset, and the output of the shallow convolutional layers of the network is extracted as shallow texture features. This feature primarily responds to the edges, colors, and texture details of the image; the output of the global pooling layer before the fully connected layer at the end of the network is extracted as the deep semantic feature. This feature characterizes the object category and scene semantics in the image, and generates global physical property features through vector concatenation operations. The calculation formula is:

[0082] ;

[0083] in, This represents a vector concatenation operation. These are weighting coefficients used to balance the contributions of texture and semantic features.

[0084] Next, facial emotion features are extracted. A multi-task cascaded convolutional neural network is used to detect faces in keyframes, locating and cropping regions of interest (ROIs). If multiple faces are detected, the largest ROI is selected as the primary target. This ROI is then input into a facial action unit (FAU) recognition network to identify subtle facial muscle movements and output a set of activation intensities for the FAUs. ,in Indicates the first The activation values ​​of each action unit are calculated, and the probability distribution of the face belonging to different basic emotion categories such as happiness, sadness, and anger is calculated using an emotion classification network. ,in Indicates the first The confidence level of an emotion is used to combine the set of action units with the emotion probability distribution to generate facial emotion features. Its numerical value is expressed as:

[0085] ;

[0086] Subsequently, spatiotemporal motion features are extracted to capture the dynamic information of the video. Candidate shot segments are treated as a continuous temporal frame sequence, and downsampling is performed to unify the frame rate. The processed frame sequence is then input into a 3D convolutional neural network, which performs convolution operations simultaneously in the spatial and temporal dimensions to extract the trajectory of human limb movements and the motion state of objects. The output vector of the global average pooling layer of the 3D convolutional neural network is extracted as the spatiotemporal motion features. .

[0087] Finally, the multimodal features are concatenated and their dimensions mapped to obtain the global physical property features from the above steps. Facial emotional features and spatiotemporal action characteristics The vectors are concatenated to form a high-dimensional hybrid visual feature vector. This is done to integrate the visual feature vector with the text feature vector. Within a comparable dimensional space, a fully connected layer is used to perform a linear transformation and dimensionality compression on the mixture vector to obtain the final visual feature vector. The calculation formula is:

[0088] ;

[0089] in, The learned projection transformation matrix, For bias vectors, This represents the transpose of the concatenated eigenvectors. This is the final output that includes the physical properties of the image, the micro-expressions of the characters, and information about their dynamic behavior.

[0090] This allows for a comprehensive analysis of the visual information of short drama shots from three dimensions: static image quality, dynamic behavior, and subtle facial expressions. It also transforms the abstract text of the script into semantic vectors that computers can understand, providing a rich and aligned data foundation for accurately judging the consistency between the shooting materials and the script requirements.

[0091] S3. Calculate the semantic similarity between the visual feature vector and the text feature vector, and normalize the semantic similarity to generate a consistency weight factor.

[0092] Furthermore, in S3, the semantic similarity between the visual feature vector and the text feature vector is calculated, specifically including:

[0093] A multimodal joint embedding space is constructed, and two independent projection transformation matrices are used to map the visual feature vectors and text feature vectors to the joint embedding space of the same dimension.

[0094] Calculate the spatial distance or cosine of the angle between the mapped visual vector and the text vector to obtain the original matching score;

[0095] The original matching scores are normalized and mapped to obtain semantic similarity values.

[0096] A consistency weight factor is generated based on the semantic similarity value. When the semantic similarity value is lower than a preset penalty threshold, the consistency weight factor is set to a value less than 1.

[0097] Specifically, first, a multimodal joint embedding space is constructed to align heterogeneous features, and then the visual feature vector output from the previous step is read. With text feature vectors As input data, two independent linear projection transformation matrices are initialized, namely the visual projection matrix. and text projection matrix The parameters of these two matrices are obtained through contrastive learning pre-training on a large-scale image-text pairing dataset. These two matrices are then used to perform matrix multiplication on the input feature vectors, mapping them to a joint embedding space of the same dimension to obtain the mapped visual vectors. and the mapped text vector The calculation process is as follows:

[0098] ;

[0099] At this point, the two vectors are in the same metric space, and their geometric distance directly reflects the similarity of their semantic content.

[0100] Next, the original matching scores between the mapped vectors are calculated, using cosine similarity as the metric. and The cosine of the angle between the two vectors measures the degree of overlap in direction. A larger value indicates a greater relevance between the visual content and the script description. This cosine value is defined as the original matching score. The calculation formula is:

[0101] ;

[0102] in, This represents the vector dot product operation. The Euclidean norm of a vector is used. Since the cosine value ranges from -1 to 1, a linear normalization method is employed to map the original matching score to the interval of 0 to 1 for easier subsequent calculations, thus obtaining the semantic similarity value. :

[0103] ;

[0104] Finally, a consistency weight factor is generated based on semantic similarity scores. The core logic of this step is to implement a penalty mechanism, that is, to reduce the weight of shots that do not match the text and images, by setting a preset penalty threshold. This threshold represents the minimum acceptable semantic matching standard, and the consistency weight factor is calculated using a piecewise function. When the semantic similarity value Below the penalty threshold When, using exponential or linear decay functions, the weighting factor is set to a value less than 1 to suppress the final score of the shot; when Higher than or equal to At this time, the weighting factor is kept at 1 or a small amount of positive incentive is given. The specific formula for calculating the consistency weighting factor is as follows:

[0105] ;

[0106] in, This is a penalty intensity coefficient; the larger the value, the heavier the penalty for low-matching lenses. This is a reward coefficient used to slightly increase the weight of high-matching shots, calculated as follows. This is the consistency weighting factor of the final output.

[0107] This effectively identifies invalid shots that, while clear in quality, deviate from the script. By using weighting factors, these shots are automatically downgraded in subsequent scoring stages, ensuring that the selected footage not only boasts high image quality but also accurately reflects the script's creative intent.

[0108] S4. Input the visual feature vector into the preset quality assessment model to output the basic quality score, and use the consistency weight factor to calculate the weighted basic quality score to obtain the comprehensive quality score of each candidate shot segment.

[0109] Furthermore, in S4, the basic quality score is weighted using a consistency weighting factor, specifically including:

[0110] The visual feature vector is input into the quality assessment network to predict the technical index score and aesthetic composition score of the image. The weighted sum is then used to output the basic quality score.

[0111] Set a baseline quality threshold. If the baseline quality score is lower than the baseline quality threshold, set the overall quality score of the candidate shot segment to zero.

[0112] If the basic quality score meets the basic quality threshold, the basic quality score is calculated by multiplication using the consistency weighting factor. When the consistency weighting factor is greater than the benchmark value, the basic quality score is increased; when the consistency weighting factor is less than the benchmark value, the basic quality score is decreased.

[0113] The calculated overall quality score is smoothed in the temporal domain to obtain the final overall quality score assigned to the candidate shot segment.

[0114] Specifically, first, the prediction calculation of the basic quality score is performed, and the visual feature vector generated in step S2 is used as the basis for the prediction. The input data is fed into a pre-built quality assessment neural network model. This model adopts a multi-task regression architecture and includes two independent output branches: a technical indicator branch that evaluates the image's sharpness, noise level, and exposure accuracy, and outputs a technical indicator score. The aesthetic composition branch evaluates the color harmony, compositional balance, and artistic sense of light and shadow in an image, outputting an aesthetic composition score. Both scores are normalized scalar values. They are combined using a weighted summation method to output the base quality score. The calculation formula is as follows:

[0115] ;

[0116] in As the weight of technical indicators, Assuming an aesthetic composition weight, and the sum of the two is 1, this step transforms abstract visual features into a quantifiable image quality score.

[0117] Next, a preliminary screening based on threshold values ​​and a consistency-weighted calculation are performed to set a basic quality threshold value. This threshold represents the minimum acceptable image quality standard in short drama production, and the calculated base quality score... Compared with this threshold value, if Values ​​below The material is directly deemed unusable, and intermediate variables are used. Forced to zero; if The value is greater than or equal to Then the consistency weight factor generated in step S3 is introduced. The score is adjusted using a multiplicative weighted approach, which involves amplifying or reducing the base score using weighting factors to obtain the adjusted intermediate quality score. The computational logic is expressed as follows:

[0118] ;

[0119] In this calculation, the baseline value is 1, when A score greater than 1 indicates a high degree of consistency between the visuals and the script, amplifying the base score. A score less than 1 indicates that the visuals deviate from the script, and the base score is compressed.

[0120] Subsequently, temporal smoothing is performed on the overall quality score. Considering that the quality scores of adjacent shots in the short drama video stream should have continuity, and to avoid abrupt changes in scores due to fluctuations in single-frame features, it is necessary to smooth the current segment by incorporating historical scoring information, thus establishing a shot sequence arranged in chronological order. Let the th... The corrected intermediate quality score of the candidate shot segments is The final score that was smoothed out in the previous moment was The final overall quality score of the current segment is calculated using the exponential moving average algorithm. :

[0121] ;

[0122] in The smoothing coefficient, ranging from 0 to 1, is used to control the degree of confidence in the current measurement value. It is calculated as follows: This is the final overall quality score assigned to the candidate shot segment.

[0123] The basic quality threshold effectively eliminates blurry, overexposed, and other unusable footage, preventing low-quality material from entering the subsequent process; the introduction of consistency weights ensures that high-scoring footage is necessarily effective shots that fit the script; and temporal smoothing eliminates random fluctuations in the scoring, making the final selection ranking results more stable and reliable.

[0124] S5. Calculate the feature distance between segments based on visual feature vectors, and cluster multiple candidate shot segments whose feature distance meets a preset threshold into the same scene set.

[0125] Furthermore, in S5, the feature distance between segments is calculated based on visual feature vectors, specifically including:

[0126] The visual feature vectors of the candidate shot segments to be compared are processed by channel weighting. The feature channel representing the background of the scene is assigned the first weight, and the feature channel representing local action changes is assigned the second weight. The value of the first weight is greater than the value of the second weight.

[0127] Calculate the distance between the feature vectors after channel weighting, and use it as the basic visual difference value;

[0128] Obtain the timestamp information of candidate shot segments and calculate the time interval, then use a preset mapping relationship to convert the time interval into a time penalty term;

[0129] The feature distance is obtained by weighting and summing the basic visual difference values ​​with the time penalty term.

[0130] Furthermore, in S5, multiple candidate shot segments whose feature distance meets a preset threshold are clustered into the same scene set, specifically including:

[0131] Initialize the list of scene sets, create the first scene set, and use the first candidate shot as the initial cluster center of the set;

[0132] Traverse subsequent candidate shot segments in chronological order and calculate the feature distance between the current segment and the cluster centers of each existing set of scenes.

[0133] The minimum value among the feature distances is selected. If the minimum value is less than the preset scene determination threshold, the current segment is assigned to the corresponding scene set, and the cluster center of the set is updated using the visual feature vector of the current segment.

[0134] If the minimum value does not meet the threshold for determining the number of sessions, a new set of sessions is created and the current segment is set as the new cluster center.

[0135] Repeat the above steps until all candidate shot segments have been traversed.

[0136] Specifically, first, channel-weighted feature vector processing and basic visual difference value calculation are performed, with the input being two candidate shot segments to be compared. and The corresponding original visual feature vectors are respectively and These two vectors are generated in step S2 and contain the physical attributes of the scene (mainly corresponding to background information) and spatiotemporal motion features (mainly corresponding to motion information). In order to focus more on identifying shots in the same scene during clustering, the different dimensional channels of the feature vector need to be weighted, and a weight mask vector with the same dimensions as the feature vector needs to be defined. Among them, the dimensional channels corresponding to the physical properties of the image (such as deep semantic features) are assigned a higher first weight. The dimension channel corresponding to the local action change is assigned a second weight with a smaller value. ,right and respectively with Perform the Hadamard product operation, which involves multiplying elements one by one to obtain the weighted eigenvector. and Then, the Euclidean distance between the two weighted vectors is calculated as the basic visual difference value. The calculation formula is as follows:

[0137] ;

[0138] in, The dimension of the feature vector. The vector represents the first time. By using this weighted processing, the visual differences in the dimensions can be kept within a small range, even if the characters' actions are different in the same scene.

[0139] Next, a time constraint is introduced to calculate the final feature distance, obtain the timestamp information of the two candidate shot segments, and calculate the absolute time interval between their center moments. The unit is seconds. To reflect the prior knowledge in short drama production that "shots in the same scene are usually close in shooting time", a nonlinear mapping function is used to convert the time interval into a time penalty term. The mapping relationship is set as a logarithmic growth function to avoid excessive numerical influence from long time intervals:

[0140] ;

[0141] in, The time sensitivity coefficient is used as the final feature distance, which measures whether two segments belong to the same scene. Finally, the basic visual difference value is added to the time penalty term to obtain the final feature distance. :

[0142] ;

[0143] Subsequently, streaming field clustering based on feature distance is performed, initializing an empty list of field sets. Read the candidate shot clip ranked first by time. Establish the first session set and will The weighted visual feature vector is directly set as the initial cluster center vector of the set. ,Right now ,at this time, .

[0144] Iterate through each subsequent candidate shot in chronological order. ( For the current segment Calculate its relationship with the existing set of all sessions. Cluster center vector Feature distance between After the traversal is complete, find the minimum value among all the calculated distances. and its corresponding set index :

[0145] ;

[0146] The minimum value Compared with the preset threshold for determining the number of sessions Compare them.

[0147] like Determine the current segment Belonging to the existing first For each set of sessions, perform the inclusion operation: Add to collection Meanwhile, the cluster centers of the set are updated using an online update strategy. To enable it to dynamically reflect the characteristics of the latest added element in the set, the update formula uses the moving average method:

[0148] ;

[0149] in, This is a historical information retention factor, typically around 0.9. This is the weighted feature vector of the current segment.

[0150] like Determine the current segment It does not belong to any existing session, representing the start of a new scene and creating a new session set. ,Will Join in, and The feature vectors are set as the cluster centers of the new set. ,Will Add to list In the process, the above traversal and judgment process is repeated until all candidate shot segments have been processed.

[0151] This achieves an automatic scene grouping mechanism that is time-sensitive and resistant to motion interference. By increasing the weight of background features, it successfully aggregates shots of different actions in the same scene, such as front shots, reverse shots, and close-ups. The introduction of a time penalty term effectively distinguishes similar scenes shot at different times, such as reshoots. The streaming clustering strategy ensures that the algorithm can build the scene structure in real time as the video stream is input, laying a structured data foundation for selecting the best shot in each independent scene.

[0152] S6. Iterate through each session set, sort the elements in the set according to the overall quality score, extract the segment with the highest score as the preferred material and output it.

[0153] Furthermore, in S6, the elements within the set are optimized and sorted based on the overall quality score, specifically including:

[0154] All candidate shot segments within the scene set are sorted in descending order according to their overall quality scores to generate an initial candidate queue;

[0155] Move the top-ranked candidate shot from the initial candidate queue to the preferred results list;

[0156] Iterate through the remaining candidate shot segments in the initial candidate queue and calculate the content repetition between the current segment and existing footage in the preferred results list.

[0157] The content repetition of the image is compared with a preset repetition threshold. If it is higher than the repetition threshold, the current segment is determined to be a repetitive segment and is removed or marked.

[0158] If the content is below the repetition threshold, the current segment is retained;

[0159] The clips in the preferred results list are filtered by duration, and clips shorter than the preset minimum usable duration are removed. The final preferred clips are then output.

[0160] Specifically, first, an initial sorting operation is performed for each individual set of sessions, and a set of sessions to be processed is selected. This includes For each candidate shot segment, read the overall quality score calculated for each segment in step S4. Based on this score, for all items in the set... The fragments are sorted in descending order of their values ​​to generate an ordered initial candidate queue. ,in The segment with the highest score, For the segment with the lowest score, after sorting, directly select the segment that ranks first in the queue, which is the segment with the highest quality. Move it to a new list and define it as the preferred results list. ,at this time .

[0161] Next, a deduplication traversal based on content repetition is performed, starting from the initial candidate queue. Starting from the second position, extract each remaining candidate shot segment in sequence. in The range of values ​​is arrive As the fragment currently to be judged, for each It needs to be calculated in relation to the list of preferred results. All existing materials in the current version The maximum content repetition between two images is determined by the Structural Similarity Index (SSIM). SSIM is a full-reference image quality assessment metric that measures the similarity between two images. It compares and analyzes images from three dimensions: brightness, contrast, and structure, reflecting the human eye's perception of image structure. In practice, it extracts... keyframe images With each material in the list keyframe images The structural similarity between the two is calculated using the following formula:

[0162] ;

[0163] in, and Images and The average brightness value represents a comparison of brightness. and The variance of the image represents the contrast ratio. The covariance of the two images represents a comparison of their structures. and To maintain computational stability and avoid zero denominators, the constant is calculated. After calculating the similarity with all elements in the list, the maximum value is taken as the content repetition rate of that segment. The calculation formula is as follows:

[0164] ;

[0165] Subsequently, a duplicate threshold comparison and filtering process is performed, and a preset duplicate threshold is set. For example, setting it to 0.85, this threshold is used to define whether the content is redundant, and the calculated value will be used to determine whether the content is redundant. and Perform a comparison, if This indicates the current segment. If a source is highly similar to a source already in the preferred list, indicating repeated shooting from the same camera position, then it is determined that... For duplicate segments, remove them or mark them as alternatives, and do not add them to the list of preferred results. ,show Even if the quality is slightly lower, if it provides new visual content, such as different shot sizes or angles, then retain the segment and add it to the list of preferred results. In the process, as the traversal proceeds, It will continue to expand, and the scope of deduplication comparison for subsequent segments will also expand accordingly.

[0166] Finally, perform time-based availability filtering and output, iterating through the list of preferred results after the above deduplication process. Check the duration attribute of each clip. Set the preset minimum available time. For example, if the time limit is set to 2 seconds, all items in the list that meet the condition will be removed. Fragmented segments, these overly short segments usually cannot carry complete narrative information and are considered invalid material. After length filtering, the final output retains the fragmented segments. The sequence of segments in the video is selected as the final preferred material for that session.

[0167] First, the quality score ensured the consistency of image quality and text in the shortlisted materials. Then, the deduplication logic built using the structural similarity index effectively eliminated duplicate and unusable footage from the same camera position. Finally, the duration filtering ensured the usability of the materials for editing, thus achieving efficient and automated conversion from original shooting materials to selected usable materials.

[0168] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically scoring and optimizing the shot quality of short drama footage, characterized in that, Includes the following steps: S1. Obtain the original short drama video stream and associated storyboard data, and use boundary detection logic to segment the original short drama video stream into independent candidate shot segments; S2. Perform image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing the physical attributes of the scene and the performance information of the characters. At the same time, parse the storyboard data to extract a text feature vector containing the scene's emotions and key action descriptions. S3. Calculate the semantic similarity value between the visual feature vector and the text feature vector, and normalize the semantic similarity value to generate a consistency weight factor. S4. Input the visual feature vector into the preset quality assessment model to output the basic quality score, and use the consistency weight factor to perform weighted calculation on the basic quality score to obtain the comprehensive quality score of each candidate shot segment. The process involves inputting the visual feature vector into a quality assessment network to predict the technical and aesthetic composition scores of the image. These scores are then weighted and summed to output a basic quality score. A basic quality threshold is set. If the basic quality score is lower than the threshold, the overall quality score of the candidate shot is set to zero. If the basic quality score meets the threshold, the basic quality score is multiplied and weighted using a consistency weighting factor. The basic quality score is increased when the consistency weighting factor is greater than a benchmark and decreased when it is less than a benchmark. The calculated overall quality score is then temporally smoothed to obtain the final overall quality score assigned to the candidate shot. S5. Calculate the feature distance between segments based on the visual feature vector, and cluster multiple candidate shot segments whose feature distance meets a preset threshold into the same scene set; The process involves channel-weighted processing of the visual feature vectors of the candidate shot segments to be compared. The feature channel representing the background is assigned a first weight, and the feature channel representing local motion changes is assigned a second weight, with the first weight being greater than the second weight. The distance between the channel-weighted feature vectors is calculated as the basic visual difference value. The timestamp information of the candidate shot segments is obtained, and the time interval is calculated. This time interval is then converted into a time penalty term using a preset mapping relationship. Finally, the basic visual difference value and the time penalty term are weighted and summed to obtain the feature distance. S6. Traverse each set of sessions, sort the elements in the set according to the comprehensive quality score, extract the segment with the highest score as the preferred material and output it. The process involves: sorting all candidate shot segments within the set of scenes in descending order of their overall quality scores to generate an initial candidate queue; moving the top-ranked candidate shot segment from the initial candidate queue into a preferred result list; iterating through the remaining candidate shot segments in the initial candidate queue and calculating the content repetition between the current segment and existing footage in the preferred result list; comparing the content repetition with a preset repetition threshold; if the repetition is higher than the threshold, the current segment is identified as a repetitive segment and is removed or marked; if the repetition is lower than the threshold, the current segment is retained; and filtering the footage in the preferred result list by duration, removing segments with a duration shorter than a preset minimum usable duration, and outputting the final preferred footage.

2. The method for automatic scoring and optimization of shot quality for short drama material according to claim 1, characterized in that, In step S1, boundary detection logic is used to segment the original short drama video stream into several independent candidate shot segments, specifically including: The original short drama video stream is processed by frame extraction at preset time intervals to obtain a continuous sequence of frame images, and each frame image in the sequence is converted from RGB color space to a color space with separate luminance and chrominance. The color distribution difference values ​​of two adjacent frames in the separated color space are calculated respectively to form an inter-frame difference sequence that varies with time. The local dynamic threshold of the inter-frame difference sequence is calculated using sliding window logic. The local dynamic threshold is composed of the sum of the mean and standard deviation of the data within the window. The elements in the inter-frame difference sequence are compared with the local dynamic threshold. When the difference value of a specific frame is greater than the local dynamic threshold and the difference value is a local maximum within a preset neighborhood range, the video stream is truncated into the candidate shot segments based on the time position of the frame.

3. The method for automatic scoring and optimization of shot quality for short drama material according to claim 1, characterized in that, In step S2, image feature extraction processing is performed on each candidate shot segment, specifically including: Keyframes are extracted from the candidate shot segments, and shallow texture features and deep semantic features of the keyframes are extracted through a neural network model and combined to generate global physical property features. Identify the facial regions of the person in the keyframe, construct the region of interest of the face, and extract the facial motion unit parameters and emotion probability distribution of the region to generate facial emotion features; The temporal frame sequence of the candidate shot fragment is input into the spatiotemporal feature extraction network to extract spatiotemporal action features; The visual feature vector is obtained by multimodal splicing and dimensional mapping of the global physical attribute features, the facial emotion features, and the spatiotemporal action features.

4. The method for automatic scoring and optimization of shot quality for short drama material according to claim 1, characterized in that, In step S3, calculating the semantic similarity value between the visual feature vector and the text feature vector specifically includes: A multimodal joint embedding space is constructed, and the visual feature vector and the text feature vector are mapped to the joint embedding space of the same dimension using two independent projection transformation matrices respectively. Calculate the spatial distance or cosine of the angle between the mapped visual vector and the text vector to obtain the original matching score; The original matching score is normalized and mapped to obtain the semantic similarity value; The consistency weight factor is generated based on the semantic similarity value, wherein when the semantic similarity value is lower than a preset penalty threshold, the consistency weight factor is set to a value less than 1.

5. The method for automatic scoring and optimization of shot quality for short drama material according to claim 1, characterized in that, In step S5, clustering multiple candidate shot segments whose feature distances satisfy a preset threshold into the same scene set specifically includes: Initialize the list of scene sets, create the first scene set, and use the first candidate shot as the initial cluster center of the set; Traverse subsequent candidate shot segments in chronological order and calculate the feature distance between the current segment and the cluster center of each existing set of scenes. The minimum value among the feature distances is selected. If the minimum value is less than the preset scene determination threshold, the current segment is assigned to the corresponding scene set, and the cluster center of the set is updated using the visual feature vector of the current segment. If the minimum value does not meet the scene determination threshold, a new scene set is created and the current segment is set as the new cluster center; Repeat the above steps until all candidate shot segments have been traversed.

Images