Shot quality automatic scoring and optimal selection method for short episode materials

Abstract

The invention relates to the technical field of multimedia data processing, in particular to an automatic shot quality scoring and optimizing method for a short episode material, which comprises the following steps of: S1, acquiring an original short episode video stream and associated split script data, and segmenting the original short episode video stream into independent candidate shot segments by utilizing boundary detection logic; s2, performing image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing picture physical attributes and character performance information, analyzing the split script data, and extracting a text feature vector containing scene emotion and key action description; according to the method, the semantic similarity of the vision and the text is calculated by constructing the multi-mode joint embedding space, the consistency weight factor is generated according to the semantic similarity, objective image technical indexes and subjective script content description are combined, and the problem that it is difficult to remove clear image quality but the content deviates from script materials through a traditional scoring method is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia data processing, and particularly relates to a short drama material shot quality automatic scoring and optimization method. BACKGROUND

[0002] With the explosive growth of the short video industry, short dramas are deeply loved by audiences because of their short production cycle and compact plot. In the actual production process, in order to capture the best performance state and scene atmosphere, the production team often repeatedly shoots the same script, resulting in a large amount of original video material. How to efficiently screen out usable shots that meet both technical standards and narrative requirements from these complex materials has become a key link to improve post-production efficiency.

[0003] Most of the existing video material optimization technologies mainly rely on traditional no-reference image quality evaluation algorithms, focusing on detecting objective technical indicators such as picture clarity, noise, exposure, and color. However, this single-dimensional evaluation method cannot understand the semantic association between video content and script, resulting in invalid materials that have clear picture quality but incorrect character performance, action deviation from script description, or inconsistency with the plot atmosphere, which cannot be effectively identified and removed. Ultimately, a large amount of manual effort is required for secondary review and screening. SUMMARY

[0004] To make up for the above shortcomings, the present application provides a short drama material shot quality automatic scoring and optimization method, aiming to improve the existing video screening technology which is limited to objective quality evaluation of underlying visual features such as picture clarity and noise, and cannot understand the semantic association between picture content and script.

[0005] The present application provides the following technical solution, a short drama material shot quality automatic scoring and optimization method, comprising the following steps:

[0006] S1, obtaining an original short drama video stream and associated script data, and using boundary detection logic to divide the original short drama video stream into independent candidate shot segments;

[0007] S2, performing image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing picture physical properties and character performance information, and simultaneously analyzing the script data to extract a text feature vector containing scene emotion and key action description;

[0008] S3, calculating the semantic similarity value between the visual feature vector and the text feature vector, and performing normalization processing on the semantic similarity value to generate a consistency weight factor;

[0009] S4, inputting the visual feature vector into a preset quality evaluation model to output a basic quality score, and weighting the basic quality score by using the consistency weight factor to obtain a comprehensive quality score of each candidate shot segment;

[0010] S5, calculating a feature distance between shot segments based on the visual feature vector, and clustering a plurality of candidate shot segments whose feature distance meets a preset threshold into a same game set;

[0011] S6, traversing each game set, preferentially sorting elements in the set according to the comprehensive quality score, and extracting a shot with the highest score as preferred material and outputting.

[0012] Preferably, in S1, the original short play video stream is segmented into a plurality of independent candidate shot segments by using boundary detection logic, specifically including:

[0013] The original short play video stream is frame-extracted at a preset time interval to obtain a continuous frame image sequence, and each frame image in the frame image sequence is converted from an RGB color space to a color space in which luminance and chrominance are separated;

[0014] Color distribution difference values of adjacent two frame images in the separated color space are calculated respectively to form a time-distributed inter-frame difference sequence;

[0015] A local dynamic threshold of the inter-frame difference sequence is calculated by using a sliding window logic, and the local dynamic threshold is composed of a sum of a mean value and a standard deviation of data in the window;

[0016] Elements in the inter-frame difference sequence are compared with the local dynamic threshold, when a difference value of a specific frame is greater than the local dynamic threshold, and the difference value is a local maximum value in a preset neighborhood range, the video stream is truncated into the candidate shot segment based on a time position of the frame.

[0017] Preferably, in S2, image feature extraction processing is performed on each candidate shot segment, specifically including:

[0018] Key frames of the candidate shot segment are extracted, shallow texture features and deep semantic features of the key frames are extracted by a neural network model, and global physical attribute features are generated by combination;

[0019] A human face region in the key frame is recognized, a face region of interest is constructed, face action unit parameters and emotion probability distribution of the region are extracted, and face emotion features are generated;

[0020] A time sequence frame sequence of the candidate shot segment is input into a space-time feature extraction network to extract space-time action features;

[0021] The global physical attribute feature, the facial emotion feature and the spatio-temporal action feature are multi-modal spliced and dimensionally mapped to obtain the visual feature vector.

[0022] Preferably, in the S3, a semantic similarity value between the visual feature vector and the text feature vector is calculated, and specifically includes:

[0023] A multi-modal joint embedding space is constructed, and two independent projection transformation matrices are used to map the visual feature vector and the text feature vector into the joint embedding space of the same dimension;

[0024] A spatial distance or a cosine value between the mapped visual vector and the text vector is calculated to obtain an original matching score;

[0025] The original matching score is normalized and mapped to obtain the semantic similarity value;

[0026] The consistency weight factor is generated based on the semantic similarity value, and when the semantic similarity value is lower than a preset penalty threshold, the consistency weight factor is set to a value less than 1.

[0027] Preferably, in the S4, the base quality score is weighted and calculated using the consistency weight factor, and specifically includes:

[0028] The visual feature vector is input into a quality evaluation network to predict a technical index score and an aesthetic composition score of the picture, and the base quality score is output after weighted summation;

[0029] A base quality threshold value is set, and if the base quality score is lower than the base quality threshold value, the comprehensive quality score of the candidate shot segment is set to zero;

[0030] If the base quality score meets the base quality threshold value, the base quality score is multiplied and weighted using the consistency weight factor, and when the consistency weight factor is greater than a reference value, the base quality score is increased, and when the consistency weight factor is less than the reference value, the base quality score is decreased;

[0031] The calculated comprehensive quality score is time-domain smoothed to obtain a final comprehensive quality score of the candidate shot segment.

[0032] Preferably, in the S5, a feature distance between the shot segments is calculated based on the visual feature vector, and specifically includes:

[0033] The visual feature vectors of the candidate shot segments to be compared are subjected to channel weighting processing. The feature channel representing the background of the scene is assigned a first weight, and the feature channel representing local action changes is assigned a second weight. The value of the first weight is greater than the value of the second weight.

[0034] Calculate the distance between the feature vectors after channel weighting, and use it as the basic visual difference value;

[0035] The timestamp information of the candidate shot segments is obtained and the time interval is calculated. The time interval is then converted into a time penalty term using a preset mapping relationship.

[0036] The feature distance is obtained by weighted summation of the basic visual difference value and the time penalty term.

[0037] Preferably, in step S5, clustering multiple candidate shot segments whose feature distance satisfies a preset threshold into the same scene set specifically includes:

[0038] Initialize the list of scene sets, create the first scene set, and use the first candidate shot as the initial cluster center of the set;

[0039] Traverse subsequent candidate shot segments in chronological order and calculate the feature distance between the current segment and the cluster center of each existing set of scenes.

[0040] The minimum value among the feature distances is selected. If the minimum value is less than the preset scene determination threshold, the current segment is assigned to the corresponding scene set, and the cluster center of the set is updated using the visual feature vector of the current segment.

[0041] If the minimum value does not meet the scene determination threshold, a new scene set is created and the current segment is set as the new cluster center;

[0042] Repeat the above steps until all candidate shot segments have been traversed.

[0043] Preferably, in step S6, the elements within the set are sorted according to the comprehensive quality score, specifically including:

[0044] All candidate shot segments within the set of scenes are sorted in descending order according to the comprehensive quality score to generate an initial candidate queue.

[0045] Move the top-ranked candidate shot in the initial candidate queue into the preferred result list;

[0046] Traverse the remaining candidate shot segments in the initial candidate queue and calculate the content repetition between the current segment and the existing material in the preferred result list.

[0047] The picture content repetition degree is compared with a preset repetition threshold, if higher than the repetition threshold, the current segment is determined as a repeated segment and is removed or marked for processing;

[0048] If lower than the repetition threshold, the current segment is retained.

[0049] The materials in the preferred result list are subjected to time length screening, segments with time length less than a preset minimum available time length are removed, and the final preferred materials are output.

[0050] The present application has the following beneficial effects:

[0051] 1. In the present application, the semantic similarity of vision and text is calculated by constructing a multi-modal joint embedding space, and a consistency weight factor is generated accordingly, which combines objective image technical indicators with subjective script content descriptions, effectively solving the problem that traditional scoring methods cannot remove high-quality but content-deviating script materials.

[0052] 2. In the present application, a feature channel differentiation weighting strategy is adopted and a time interval is introduced as a penalty term, which weakens the interference of character actions while enhancing the recognition ability of scene background, realizing the automatic clustering of shots that are not continuous in time but are shot in the same scene into independent scene sets.

[0053] 3. In the present application, the picture content repetition degree of high-score materials in the preferred queue is detected by using structural similarity index, which removes redundant shots in the same position under the premise of ensuring that the material quality meets the requirements, realizing the automatic preliminary screening of short drama materials from original data to editing available sequences. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart of a short drama material shot quality automatic scoring and optimization method according to the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings, obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0056] The present application provides a short drama material shot quality automatic scoring and optimization method, as shown in Figure 1 The method comprises the following steps:

[0057] S1, acquire the original short drama video stream and the associated screenplay data, and use boundary detection logic to divide the original short drama video stream into independent candidate shot segments;

[0058] Further, in S1, the original short drama video stream is segmented into several independent candidate shot segments by using boundary detection logic, specifically including:

[0059] Frame extraction processing is performed on the original short drama video stream at a preset time interval to obtain a sequence of continuous frame images, and each frame image in the sequence of frame images is converted from an RGB color space to a color space with luminance and chrominance separation;

[0060] Color distribution difference values of adjacent two frame images in the separated color space are calculated respectively to form a sequence of inter-frame difference values distributed over time;

[0061] A local dynamic threshold of the sequence of inter-frame difference values is calculated by using sliding window logic, and the local dynamic threshold is composed of the sum of the mean value and the standard deviation of the data in the window;

[0062] Elements in the sequence of inter-frame difference values are compared with the local dynamic threshold, when the difference value of a specific frame is greater than the local dynamic threshold, and the difference value is a local maximum value in a preset neighborhood range, the video stream is truncated into a candidate shot segment based on the time position of the frame.

[0063] Specifically, first, frame extraction and color space conversion processing of the video stream are performed, and the input original short drama video stream is defined as Frame extraction and color space conversion processing of the video stream are performed, and the input original short drama video stream is defined as , discrete sampling is performed on at a preset time interval (e.g. every 0.5 seconds) to obtain a sequence of continuous frame images in time, where is the total number of frames, in order to reduce the overall fluctuation of light intensity to interfere with content detection, each frame image in the sequence is converted from the original RGB color space to a color space with luminance and chrominance separation, such as YUV or CIELAB color space, taking conversion to YUV space as an example, let be the pixel value of the coordinate in the image, the luminance component and the chrominance component after conversion are calculated by a linear transformation matrix, at this time, the feature representation of each frame image in the separated color space is , which retains the structured information of the image and decouples the luminance and color.

[0064] Next, the color distribution difference between adjacent frames is calculated, for adjacent two frames and in the sequence, the pixel value difference in the luminance channel and the chrominance channel is calculated respectively, thereby generating a quantitative difference value It is represented by the sum of the absolute values ​​of the pixel differences between each channel, and its calculation formula is as follows:

[0065] ;

[0066] in, and These are the height and width of the image, respectively. These are the weighting coefficients for the corresponding channels, typically the luminance channel. The weight is set higher than that of the chroma channel. Representing the individual channel components in the separate color space, an inter-frame difference sequence is formed by traversing and calculating the entire sequence, thus generating a time-varying sequence. This sequence reflects the degree of dynamic change in the video content.

[0067] Subsequently, a local dynamic threshold is constructed to adapt to the rhythmic changes of the video content, using a threshold of length [missing information]. The sliding window in the inter-frame difference sequence Swipe up to view the difference value at the current time. Its corresponding local dynamic threshold Based on the statistical characteristics of the data within the window, the set of differences within the window is calculated first. mean and standard deviation The calculation formula is as follows:

[0068] ;

[0069] Based on the above statistics, a local dynamic threshold is defined. A linear combination of the mean and standard deviation:

[0070] ;

[0071] in, This is the sensitivity adjustment coefficient, usually set to 1. This threshold setting method can automatically adjust the judgment standard according to the intensity of the current video segment, automatically increasing the threshold in intense fighting scenes and automatically decreasing the threshold in static dialogue scenes.

[0072] Finally, boundary determination and video stream truncation are performed, separating each element in the inter-frame difference sequence. Its corresponding local dynamic threshold To determine whether a frame position is a shot transition boundary through a comparison, two conditions must be met simultaneously: firstly, the current difference value... The value must be strictly greater than the local dynamic threshold. ,Right now Secondly, the difference value It must be a local maximum within a preset neighborhood, that is, it must satisfy... and When all of the above conditions are met, the first... Frame and the There are shot transitions between frames; these points in time are recorded as breakpoints. The original short drama video stream is then reconstructed using all identified breakpoints. Physically truncate the timeline to output a series of independent candidate shot fragments. .

[0073] This effectively overcomes the shortcomings of a single fixed threshold that cannot take into account the fast and slow rhythm switching of short dramas. It reduces false detections caused by lighting flicker by using luminance-color separation, and suppresses interference caused by rapid object movement by using local dynamic thresholds and local maxima, thereby achieving high-precision automated shot segmentation of short drama materials.

[0074] S2. Perform image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing the physical attributes of the image and the performance information of the characters. At the same time, parse the storyboard data to extract a text feature vector containing the scene's emotions and key action descriptions.

[0075] Furthermore, in S2, image feature extraction processing is performed on each candidate shot segment, specifically including:

[0076] Keyframes are extracted from candidate shot segments. Shallow texture features and deep semantic features of the keyframes are extracted through a neural network model and combined to generate global physical property features.

[0077] Identify the facial regions of people in keyframes, construct the region of interest for the face, and extract the facial motion unit parameters and emotion probability distribution of the region to generate facial emotion features.

[0078] The temporal frame sequence of the candidate shot fragments is input into the spatiotemporal feature extraction network to extract spatiotemporal action features;

[0079] The global physical attribute features, facial emotion features, and spatiotemporal action features are multimodal spliced ​​and dimensionally mapped to obtain the visual feature vector.

[0080] Specifically, first, text feature extraction is performed for the screenplay data, screenplay data associated with the current candidate shot segment in time or scene number is obtained, which contains adjectival descriptions of scene atmosphere, psychological activity descriptions of characters, and text descriptions of key body movements, the text content is preprocessed by word segmentation and stop word removal to obtain a normalized word sequence, the word sequence is input into a pre-trained transformer bidirectional encoding representation model, i.e. BERT model, the multi-layer self-attention mechanism of the model is used to capture the context semantic association in the text, and the vector corresponding to the classification identification bit in the output layer of the model for representing the overall semantic of the sentence is defined as the text feature vector The vector numerically represents the scene emotional tone and key action requirements of the script for the current shot.

[0081] Secondly, image key frame extraction and global physical attribute feature construction of the candidate shot segment are performed, the duration information of the candidate shot segment is read, the frame image at the middle time of the segment is selected as the key frame representing the segment, the key frame is input into a deep convolutional neural network model, such as a residual network pre-trained on the ImageNet dataset, the output of the shallow convolutional layer of the network is extracted as the shallow texture feature The feature mainly responds to the edges, colors and texture details of the image; the output of the global pooling layer before the end full connection layer of the network is extracted as the deep semantic feature The feature represents the object categories and scene semantics in the image, and the global physical attribute feature is generated by vector splicing operation The calculation formula is:

[0082] ;

[0083] Among them, represents the vector splicing operation, is a weight coefficient for balancing the contribution degree of texture and semantic features.

[0084] Then, the face emotion feature is extracted, the multi-task cascading convolutional neural network is used for face detection in the key frame, the face region of interest is located and cropped, if multiple faces are detected, the face region with the largest area is selected as the main target, the face region of interest is input into the face action unit recognition network, the micro-movement of facial muscles is recognized, and the activation intensity set of the face action unit is output Among them, represents the activation value of the th action unit, and the emotion classification network is used to calculate the probability distribution of the face belonging to different basic emotion categories such as happiness, sadness and anger Among them, represents the activation value of the A confidence of the emotion, combining the action unit set with the emotion probability distribution to generate a facial emotion feature , the value of which is represented as:

[0085] ;

[0086] Subsequently, a spatio-temporal action feature is extracted to capture dynamic information of the video, the candidate shot segment is taken as a continuous time sequence frame sequence, down-sampling processing is performed to unify the frame number, and the processed frame sequence is input into a three-dimensional convolutional neural network, which simultaneously performs convolution operation in the spatial dimension and the time dimension to extract the human body limb motion trajectory and the object motion state, and an output vector of a global average pooling layer of the three-dimensional convolutional neural network is extracted as the spatio-temporal action feature .

[0087] Finally, the splicing and dimension mapping of the multi-modal feature are performed, the global physical attribute feature , the facial emotion feature , and the spatio-temporal action feature obtained in the above steps are concatenated to form a high-dimensional mixed visual feature vector, in order to make the visual feature vector and the text feature vector be in the same dimension space for comparison, a full connection layer is used to perform linear transformation and dimension compression on the mixed vector to obtain the final visual feature vector , and the calculation formula is:

[0088] ;

[0089] wherein, is a learned projection transformation matrix, is a bias vector, denotes the transpose of the feature vector after splicing, which is the final output result containing the picture physical attribute, the human micro-expression, and the dynamic behavior information.

[0090] In this way, the visual information of the short drama shot can be comprehensively analyzed from three dimensions of static quality, dynamic behavior, and subtle expression, and the abstract text of the script can be converted into a semantic vector understandable by a computer, thereby providing a rich and aligned data basis for subsequent accurate judgment of the consistency between the shooting materials and the script requirements.

[0091] S3, calculating a semantic similarity value between the visual feature vector and the text feature vector, and performing normalization processing on the semantic similarity value to generate a consistency weight factor;

[0092] Further, in S3, the semantic similarity value between the visual feature vector and the text feature vector is calculated, and specifically includes:

[0093] A multi-modal joint embedding space is constructed, and two independent projection transformation matrices are used to map the visual feature vector and the text feature vector into the same dimension joint embedding space respectively;

[0094] The spatial distance or the cosine value between the mapped visual vector and the text vector is calculated to obtain an original matching score;

[0095] The original matching score is normalized and mapped to obtain a semantic similarity value;

[0096] A consistency weight factor is generated based on the semantic similarity value, wherein when the semantic similarity value is lower than a preset penalty threshold, the consistency weight factor is set to a value less than 1.

[0097] Specifically, first, a multi-modal joint embedding space is constructed to realize the alignment of heterogeneous features, and the visual feature vector and the text feature vector output by the previous step are read as input data. Two independent linear projection transformation matrices are initialized, namely the visual projection matrix and the text projection matrix The parameters of the two matrices are obtained by contrastive learning pre-training on a large-scale image-text pair dataset. The two matrices are used to perform matrix multiplication on the input feature vectors to map them into the same dimension joint embedding space, obtaining the mapped visual vector and the mapped text vector The calculation process is as follows:

[0098] ;

[0099] At this time, the two vectors are in the same metric space, and their geometric distance directly reflects the similarity of the semantic content.

[0100] Next, the original matching score between the mapped vectors is calculated, and the cosine similarity is used as the measurement standard to calculate the cosine value between and The value measures the degree of coincidence of the two vectors in the direction, and the larger the value, the more relevant the picture content and the script description. The cosine value is defined as the original matching score The calculation formula is:

[0101] ;

[0102] Wherein, represents the vector dot product operation, Euclidean norm of a vector, since the cosine value ranges from -1 to 1, in order to facilitate subsequent calculation, the original matching score is mapped to the interval of 0 to 1 by using linear normalization method, and the semantic similarity value is obtained :

[0103] ;

[0104] Finally, a consistency weight factor is generated based on the semantic similarity value, the core logic of this step is to implement a penalty mechanism, that is, to give a weight reduction processing to the shot with inconsistent text and picture, and a preset penalty threshold is set , which represents the minimum acceptable semantic matching standard, and the consistency weight factor is calculated by using a piecewise function When the semantic similarity value is lower than the penalty threshold , the weight factor is set to a value less than 1 by using an exponential decay or linear decay function to suppress the final score of the shot; when is higher than or equal to , the weight factor is kept at 1 or given a slight positive incentive, and the specific consistency weight factor calculation formula is as follows:

[0105] ;

[0106] Wherein, is the penalty intensity coefficient, the greater the value, the heavier the penalty to the low matching degree shot, is the reward coefficient, which is used to slightly increase the weight of the high matching degree shot, and the calculated is the final output consistency weight factor.

[0107] In this way, the invalid shot with clear picture but deviating content from the script can be effectively identified, and the weight factor is automatically reduced in the subsequent scoring link, so as to ensure that the selected material not only has high picture quality, but also can accurately restore the creative intention of the script.

[0108] S4, input the visual feature vector into the preset quality evaluation model to output a basic quality score, and weight the basic quality score by using the consistency weight factor to obtain a comprehensive quality score of each candidate shot segment;

[0109] Further, in S4, the basic quality score is weighted by using the consistency weight factor, specifically including:

[0110] The visual feature vector is input into the quality evaluation network to predict the technical index score and aesthetic composition score of the picture, and the basic quality score is output after weighted summation;

[0111] A base quality threshold is set, and if the base quality score is lower than the base quality threshold, the comprehensive quality score of the candidate shot segment is set to zero;

[0112] If the base quality score meets the base quality threshold, the base quality score is multiplied by a consistency weight factor for weighted calculation, and when the consistency weight factor is greater than a reference value, the base quality score is increased, and when the consistency weight factor is less than the reference value, the base quality score is decreased;

[0113] The calculated comprehensive quality score is subjected to time domain smoothing processing to obtain the final comprehensive quality score assigned to the candidate shot segment.

[0114] Specifically, first, the prediction calculation of the base quality score is performed, and the visual feature vector generated in step S2 is input into the preset quality evaluation neural network model. As input data, the preset quality evaluation neural network model is input, which adopts a multi-task regression architecture and includes two independent output branches: a technical index branch for evaluating the sharpness, noise level and exposure accuracy of the picture, outputting a technical index score ; and an aesthetic composition branch for evaluating the color harmony, composition balance and light and shadow artistic sense of the picture, outputting an aesthetic composition score Both scores are normalized scalar values, and the two scores are combined by weighted summation to output the base quality score , and the calculation formula is as follows:

[0115] ;

[0116] Wherein is the technical index weight, is the aesthetic composition weight, and the sum of the two is 1. This step converts abstract visual features into quantitative picture quality scores.

[0117] Then, threshold-based preliminary screening and consistency weighting calculation are performed, and a base quality threshold is set, which represents the minimum acceptable picture quality standard in short video production. The calculated base quality score is compared with the threshold, and if the value is lower than , it is directly determined that the material is unusable, and the intermediate variable is forced to zero. If the value is greater than or equal to , a consistency weight factor generated in step S3 is introduced to modify it, and the modification logic is multiplication weighting, that is, the base score is amplified or reduced by using the weight factor to obtain the modified intermediate quality score , and the calculation logic is expressed as follows:

[0118] ;

[0119] In this calculation, the reference value is 1, when greater than 1, it indicates that the picture is highly consistent with the script, and the basic score is amplified, when less than 1, it indicates that the picture deviates from the script, and the basic score is compressed.

[0120] Then, the time domain smoothing processing is performed on the comprehensive quality score, considering that the quality scores of adjacent shots in the short video stream should have continuity, avoiding the score mutation caused by single frame feature fluctuation, the current segment needs to be smoothed combined with the historical score information, a shot sequence arranged in time sequence is established, and the modified intermediate quality score of the m-th candidate shot segment is denoted as , the final score of the last time smoothed is denoted as , and the final comprehensive quality score of the current segment is calculated by using the exponential moving average algorithm :

[0121] ;

[0122] Wherein is a smoothing coefficient, the value range is between 0 and 1, which is used to control the degree of trust in the current measurement value, and the calculated is the final comprehensive quality score given to the candidate shot segment.

[0123] The basic quality threshold value effectively eliminates the waste pieces such as blur and overexposure, preventing low-quality materials from entering the subsequent process; the introduction of the consistency weight ensures that the high-score materials are effective shots that fit the script, and the time domain smoothing processing eliminates the random jitter of the score, making the final optimal sorting result more stable and reliable.

[0124] S5, calculate the feature distance between the segments based on the visual feature vector, and cluster the multiple candidate shot segments whose feature distance meets the preset threshold value into the same set of scenes;

[0125] Further, in S5, the feature distance between the segments is calculated based on the visual feature vector, specifically including:

[0126] The visual feature vectors of the candidate shot segments to be compared are subjected to channel weighting processing, wherein the feature channels representing the picture background are assigned a first weight, and the feature channels representing the local action change are assigned a second weight, and the value of the first weight is greater than the value of the second weight;

[0127] The distance between the feature vectors after channel weighting processing is calculated as the basic visual difference value;

[0128] ​Timestamp information of the candidate shot segments is acquired and a time interval is calculated, the time interval is converted into a time penalty term by using a preset mapping relationship;

[0129] The basic visual difference value and the time penalty term are weighted and summed to obtain a feature distance.

[0130] Further, in S5, the plurality of candidate shot segments whose feature distances satisfy a preset threshold are clustered into a same game set, and the clustering specifically comprises:

[0131] A game set list is initialized, a first game set is established, and a first candidate shot segment is taken as an initial clustering center of the set;

[0132] The subsequent candidate shot segments are traversed in time sequence, and the feature distance between the current segment and the clustering center of each existing game set is calculated respectively;

[0133] The minimum value in the feature distance is selected, if the minimum value is less than a preset game determination threshold, the current segment is classified into the corresponding game set, and the clustering center of the set is updated by using the visual feature vector of the current segment;

[0134] If the minimum value does not satisfy the game determination threshold, a new game set is created and the current segment is set as a new clustering center;

[0135] The above steps are repeatedly executed until all candidate shot segments are traversed.

[0136] Specifically, first, the feature vector processing based on channel weighting and the basic visual difference value calculation are performed, and the input is two candidate shot segments to be compared and The corresponding original visual feature vectors of the two segments are respectively and The two vectors are generated in S2 step, and contain picture physical properties (mainly corresponding to background information) and space-time action features (mainly corresponding to action information), in order to pay more attention to identifying the shots in the same scene during clustering, different dimension channels of the feature vector need to be weighted, and a weight mask vector which has the same dimension as the feature vector is defined, wherein the dimension channel corresponding to the picture physical property (such as the deep semantic feature) is given a first weight with a larger value, the dimension channel corresponding to the local action change is given a second weight with a smaller value, and are multiplied by Hadamard product operation, i.e. element by element, to obtain the weighted feature vectors and , then, the Euclidean distance between the two weighted vectors is calculated as the basic visual difference value , the formula is as follows:

[0137] ;

[0138] wherein, is the dimension of the feature vector, represents the numerical value of the vector in the dimension, through this weighted processing, even if the same scene under different actions, the visual difference value will be controlled in a small range.

[0139] Next, introduce time constraint to calculate the final feature distance, get the timestamp information of the two candidate shot segments, calculate the absolute time interval of their center time , unit: seconds, in order to reflect the priori knowledge of "the same scene shot is usually adjacent in shooting time" in short drama production, the time interval is converted into time penalty term by using a nonlinear mapping function, the mapping relationship is set as a logarithmic growth function to avoid the influence of too large value caused by long time interval:

[0140] ;

[0141] wherein, is the time sensitivity coefficient, finally, the basic visual difference value and the time penalty term are added to obtain the final feature distance measuring whether the two segments belong to the same scene:

[0142] ;

[0143] Then, perform feature distance-based streaming scene clustering, initialize an empty scene set list , read the candidate shot segment with the first time in time sequence , establish the first scene set , and set the weighted visual feature vector of as the initial clustering center vector of the set , that is , at this time, .

[0144] According to time sequence, each subsequent candidate shot segment is traversed in turn ( ), for the current segment , the feature distance between it and the clustering center vector of all existing scene sets is calculated respectively After traversal, the minimum value is found from all calculated distances and its corresponding episode set index :

[0145] ;

[0146] comparing the minimum value with a preset episode decision threshold .

[0147] If , it is determined that the current segment belongs to an existing episode set , and a grouping operation is performed: adding to the set , and updating the cluster center of the set using an online updating strategy to dynamically reflect the characteristics of the latest added elements in the set, and the update formula uses a moving average method:

[0148] ;

[0149] wherein is a historical information retention coefficient, usually taking about 0.9, is the weighted feature vector of the current segment.

[0150] If , it is determined that the current segment does not belong to any existing episode, representing the beginning of a new scene, a new episode set is created, and is added to it, and the feature vector of is set as the cluster center of the new set , and is added to the list , and the above traversal and determination process is repeated until all candidate shot segments are processed.

[0151] In this way, an automatic episode grouping mechanism sensitive to time and resistant to action interference is realized, and by increasing the background feature weight, different actions such as positive fighting, negative fighting and close-up shots in the same scene are successfully aggregated together, and the introduction of the time penalty term effectively distinguishes similar scenes shot at different times, such as retakes; the streaming clustering strategy ensures that the algorithm can construct the episode structure in real time as the video stream is input, laying a structured data foundation for subsequent optimization of the best shot in each independent episode.

[0152] S6, traverse each episode set, and perform optimal sorting of the elements in the set according to the comprehensive quality score, extract the highest-scoring segment as the optimal material and output.

[0153] ​Further, in S6, the elements in the set are sorted according to the comprehensive quality score, specifically including:

[0154] All candidate shot segments in the game set are arranged in descending order according to the comprehensive quality score to generate an initial candidate queue;

[0155] The candidate shot segment ranked first in the initial candidate queue is moved to the preferred result list;

[0156] The remaining candidate shot segments in the initial candidate queue are traversed, and the picture content repetition degree between the current segment and the existing materials in the preferred result list is calculated in turn;

[0157] The picture content repetition degree is compared with the preset repetition threshold value, if higher than the repetition threshold value, the current segment is determined as a repeated segment and is removed or marked;

[0158] If lower than the repetition threshold value, the current segment is retained;

[0159] The materials in the preferred result list are screened by time length, and the segments with time length less than the preset minimum available time length are removed, and the final preferred materials are output.

[0160] Specifically, first, for each independent game set, an initial sorting operation is performed, and a to-be-processed game set is selected , which contains candidate shot segments, the comprehensive quality score calculated for each segment in step S4 is read , and all segments in the set are arranged in descending order according to the score value from large to small, thereby generating an ordered initial candidate queue , wherein is the highest score segment, is the lowest score segment, after sorting, the segment ranked first in the queue, i.e. the highest quality segment is directly moved to a new list, defined as the preferred result list , at this time .

[0161] Then, the content repetition degree-based deduplication traversal is performed, starting from the second position of the initial candidate queue , each remaining candidate shot segment is extracted in turn , wherein , the value range is to , as the current to-be-determined segment, for each , the picture content repetition degree between it and all existing materials in the preferred result list is calculated The maximum picture content repetition degree between the two, where the structural similarity index, namely SSIM, is used as the repetition degree measurement standard. The structural similarity index is a full-reference image quality evaluation index for measuring the similarity of two images. It compares images from three dimensions of brightness, contrast and structure, and can reflect the human eye's perception of image structure. In the specific implementation, the key frame images of the two lists are extracted respectively The key frame images of each material in the list The structural similarity values of the two are calculated, and the calculation formula is:

[0162] ;

[0163] Wherein, and are the average brightness values of images and , representing the comparison of brightness, and are the variances of the images, representing the comparison of contrast, is the covariance of the two images, representing the comparison of structure, and are constants for maintaining calculation stability and avoiding zero denominator, and the similarity of all elements in the list is calculated, and the maximum value is taken as the picture content repetition degree of the segment , and the calculation formula is:

[0164] ;

[0165] Subsequently, the repetition threshold comparison and screening are performed. A preset repetition threshold , for example, is set to 0.85. The threshold is used to define whether the content is redundant. The calculated is compared with . If , it indicates that the current segment is highly similar to a certain material in the preferred list, that is, there is a same location repeated shooting, and the is determined as a repeated segment, which is excluded or marked as an alternative and not added to the preferred result list. If , it indicates that Although the quality is slightly lower, it provides new visual content, that is, provides different views or angles, so the segment is retained and added to the preferred result list . With the traversal, the will be continuously expanded, and the range of the subsequent segment repetition comparison will also be expanded.

[0166] ​​Finally, perform time-based availability filtering and output, iterating through the list of preferred results after the above deduplication process. Check the duration attribute of each clip. Set the preset minimum available time. For example, if the time limit is set to 2 seconds, all items in the list that meet the condition will be removed. Fragmented segments, these overly short segments usually cannot carry complete narrative information and are considered invalid material. After length filtering, the final output retains... The sequence of segments in the video is selected as the final preferred material for that session.

[0167] First, the quality score ensured the consistency of image quality and text in the shortlisted materials. Then, the deduplication logic built using the structural similarity index effectively eliminated duplicate and unusable footage from the same camera position. Finally, the duration filtering ensured the usability of the materials for editing, thus achieving efficient and automated conversion from original shooting materials to selected usable materials.

[0168] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically scoring and optimizing the shot quality of short drama footage, characterized in that, The method comprises the following steps: S1, obtaining an original short play video stream and associated screenplay data, and using boundary detection logic to divide the original short play video stream into independent candidate shot segments; S2, performing image feature extraction processing on each candidate shot segment to obtain a visual feature vector containing picture physical properties and character performance information, and simultaneously analyzing the screenplay data to extract a text feature vector containing scene emotion and key action descriptions; S3, calculating the semantic similarity value between the visual feature vector and the text feature vector, and performing normalization processing on the semantic similarity value to generate a consistency weight factor; S4, inputting the visual feature vector into a preset quality evaluation model to output a basic quality score, and using the consistency weight factor to perform weighted calculation on the basic quality score to obtain a comprehensive quality score of each candidate shot segment; S5, calculating the feature distance between segments based on the visual feature vector, and clustering multiple candidate shot segments whose feature distance meets a preset threshold value into a same set of scenes; S6, traversing each set of scenes, and performing optimal sorting on the elements in the set according to the comprehensive quality score to extract the highest-scoring segment as the preferred material and output it.

2. The method of claim 1, wherein the method further comprises: determining a quality of each of the plurality of shots; and selecting a shot of the plurality of shots based on the determined quality of each of the plurality of shots. In S1, the original short play video stream is divided into several independent candidate shot segments by using boundary detection logic, specifically including: frame extraction processing is performed on the original short play video stream at a preset time interval to obtain a continuous frame image sequence, and each frame image in the frame image sequence is converted from an RGB color space to a color space in which luminance and chrominance are separated; color distribution difference values of adjacent two frame images in the separated color space are calculated respectively to form a frame difference sequence distributed over time; a local dynamic threshold value of the frame difference sequence is calculated by using a sliding window logic, and the local dynamic threshold value is composed of the sum of the mean value and the standard deviation of the data in the window; elements in the frame difference sequence are compared with the local dynamic threshold value, when the difference value of a specific frame is greater than the local dynamic threshold value, and the difference value is a local maximum value in a preset neighborhood range, the video stream is truncated into the candidate shot segment based on the time position of the frame.

3. The method of claim 1, wherein the method further comprises: determining a quality of each of the plurality of shots; and selecting a shot of the plurality of shots based on the determined quality of each of the plurality of shots. In S2, image feature extraction processing is performed on each candidate shot segment, specifically including: key frame extraction is performed on the candidate shot segment, shallow texture features and deep semantic features of the key frame are extracted by a neural network model, and global physical property features are generated by combination; a human face region in the key frame is recognized, a face region of interest is constructed, face action unit parameters and emotion probability distribution in the region are extracted, and face emotion features are generated; a time-space feature extraction network is inputted with a time sequence frame sequence of the candidate shot segment to extract time-space action features; the global physical property features, the face emotion features and the time-space action features are subjected to multi-modal splicing and dimension mapping to obtain the visual feature vector.

4. The method of claim 1, wherein the method further comprises: determining a quality of each of the plurality of shots; and selecting a shot of the plurality of shots based on the determined quality of each of the plurality of shots. In S3, the semantic similarity value between the visual feature vector and the text feature vector is calculated, specifically including: A multi-modal joint embedding space is constructed, and two independent projection transformation matrices are used to map the visual feature vector and the text feature vector into the joint embedding space with the same dimension. The spatial distance or the cosine value between the mapped visual vector and the text vector is calculated to obtain an original matching score. The original matching score is normalized to obtain a semantic similarity value. The consistency weight factor is generated based on the semantic similarity value, and when the semantic similarity value is lower than a preset penalty threshold, the consistency weight factor is set to a value less than 1.

5. The method of claim 1, wherein the method further comprises: determining a quality of each of the plurality of shots; and selecting a shot of the plurality of shots based on the determined quality of each of the plurality of shots. In S4, the base quality score is weighted using the consistency weight factor, specifically including: The visual feature vector is input into a quality evaluation network to predict the technical index score and the aesthetic composition score of the picture, and the base quality score is output after weighted summation; A base quality threshold is set, and if the base quality score is lower than the base quality threshold, the comprehensive quality score of the candidate shot segment is set to zero; If the base quality score meets the base quality threshold, the base quality score is multiplied by the consistency weight factor, and when the consistency weight factor is greater than a reference value, the base quality score is increased, and when the consistency weight factor is less than the reference value, the base quality score is decreased; The calculated comprehensive quality score is time-domain smoothed to obtain the final comprehensive quality score of the candidate shot segment.

6. The method of claim 1, wherein the method further comprises: In S5, the feature distance between segments is calculated based on the visual feature vector, specifically including: The visual feature vector of the candidate shot segment to be compared is subjected to channel weighting processing, wherein the feature channels representing the picture background are assigned a first weight, and the feature channels representing local motion changes are assigned a second weight, and the value of the first weight is greater than the value of the second weight; The distance between the channel-weighted feature vectors is calculated as a base visual difference value; The timestamp information of the candidate shot segment is obtained and the time interval is calculated, and the time interval is converted into a time penalty term using a preset mapping relationship; The base visual difference value and the time penalty term are weighted and summed to obtain the feature distance.

7. The method of claim 1, wherein the method further comprises: determining a quality of each of the plurality of shots; and selecting a shot of the plurality of shots based on the determined quality of each of the plurality of shots. In S5, the candidate shot segments whose feature distances meet a preset threshold are clustered into the same session set, specifically including: Initializing a session set list, establishing a first session set, and setting the first candidate shot segment as the initial clustering center of the set; The subsequent candidate shot segments are traversed in time sequence, and the feature distance between the current segment and the clustering center of each existing session set is calculated; The minimum value in the feature distance is selected, and if the minimum value is less than a preset session determination threshold, the current segment is assigned to the corresponding session set, and the clustering center of the set is updated using the visual feature vector of the current segment; If the minimum value does not meet the session determination threshold, a new session set is created and the current segment is set as the new clustering center; The above steps are repeated until all candidate shot segments are traversed.

8. The method of claim 1, wherein the method further comprises: determining a quality of each of the plurality of shots; and selecting a shot of the plurality of shots based on the determined quality of each of the plurality of shots. In the S6, the elements in the set are preferentially sorted according to the comprehensive quality score, specifically comprising: All candidate shot segments in the set of matches are arranged in descending order of the comprehensive quality score to generate an initial candidate queue; The first-ranked candidate shot segment in the initial candidate queue is moved to the preferred result list; The remaining candidate shot segments in the initial candidate queue are traversed, and the picture content repetition degree between the current segment and the existing materials in the preferred result list is calculated in turn; The picture content repetition degree is compared with the preset repetition threshold value, if higher than the repetition threshold value, the current segment is determined as a repeated segment and is subjected to rejection or marking processing; If lower than the repetition threshold value, the current segment is retained; The materials in the preferred result list are subjected to time length screening, and the segments with time length less than the preset minimum available time length are rejected, and the final preferred materials are output.

Images