Video program feature extraction method based on artificial intelligence

Abstract

The invention provides a video program feature extraction method based on artificial intelligence, relates to the technical field of IPTV, is applied to a cloud node, and comprises the following steps: reading an IPTV video file stored in the cloud node, and decoding the IPTV video file into an original frame sequence; performing shot disassembly on the original frame sequence to determine a plurality of shot segments; carrying out adaptive key frame extraction on each shot segment, determining sub-segments by taking the key frames as cores, and carrying out differentiated extraction on image features of each frame of image in the sub-segments to obtain feature subsets corresponding to the sub-segments; integrating the feature subsets belonging to the same shot segment to obtain a scene feature set; and integrating the scene feature set corresponding to each shot segment to obtain a video feature set corresponding to the IPTV video file. According to the scheme, a new feature extraction scheme is set, a novel IPTV architecture is adapted, and the extraction efficiency of the IPTV video program feature set is improved.

Description

Technical Field

[0001] The present application relates to the field of IPTV technology, and in particular to a method for extracting video program features based on artificial intelligence. Background Art

[0002] In existing IPTV architectures, various ancillary services are typically delivered in the cloud and embedded within video programs. This approach is inflexible and difficult to dynamically expand or adjust. If new features are to be added (such as intelligent analysis of characters and objects in a program, or smart advertising recommendations), existing solutions, which are fixed to the encoding process, are clearly inflexible and difficult to apply. Therefore, we propose a new IPTV architecture that synchronizes the video stream with the feature stream: the video stream is sent to the end-side node for decoding; the feature stream is sent to the edge node, which uses it for intelligent processing and recognition to provide ancillary services. The end-side node can then integrate the ancillary services with the video content for display. This allows for flexible expansion of ancillary services and meets users' real-time interactive needs. (This technology has been used in another patent application.)

[0003] To adapt to the auxiliary services within this architecture, the large number of video programs stored in the cloud requires feature extraction to support pre-encoding of feature streams. Although some video programs have undergone feature extraction to support various intelligent applications (such as intelligent recommendations and highlight clip editing), direct application is difficult. This is because the designed IPTV architecture (a network of client-side nodes, edge nodes, and cloud nodes) requires feature streams that correspond one-to-one with the video program's frames. Traditional feature extraction solutions do not meet these requirements. Therefore, a targeted feature extraction solution is needed to integrate with the IPTV architecture and provide a foundation for the flexible deployment of more intelligent auxiliary services.

[0004] However, directly using existing feature extraction solutions to extract full feature vectors frame by frame for IPTV video programs is prohibitively expensive. Cloud nodes currently store a vast number of video programs, each containing a large number of frames (for example, a 100-minute movie at 30 frames per second would have 6,000 x 30 = 180,000 frames). Extracting full features frame by frame consumes excessive resources and is inefficient. Therefore, a more efficient feature extraction solution suitable for this IPTV architecture is needed. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a video program feature extraction method based on artificial intelligence to improve the extraction efficiency of IPTV video program feature streams and adapt to the new IPTV architecture.

[0006] In order to achieve the above objectives, the embodiments of the present application are implemented in the following manner: In the first aspect, an embodiment of the present application provides an artificial intelligence-based video program feature extraction method, which is applied to a cloud node, including: reading an IPTV video file stored in the cloud node and decoding it into an original frame sequence; performing lens decomposition on the original frame sequence to determine multiple lens segments; performing adaptive key frame extraction on each lens segment, and determining a sub-segment with the key frame as the core, and then differentially extracting the image features of each frame image in the sub-segment to obtain a feature subset corresponding to the sub-segment; integrating the feature subsets belonging to the same lens segment to obtain a scene feature set; integrating the scene feature set corresponding to each lens segment to obtain a video feature set corresponding to the IPTV video file.

[0007] Beneficial Effects: This solution reads IPTV video files stored on cloud nodes, decodes them into raw frame sequences, performs lens decomposition to determine multiple lens segments, then performs adaptive keyframe extraction on each lens segment, and determines sub-segments with the keyframes as the core. The image features of each frame in the sub-segment are then differentially extracted (full feature vector extraction is performed for keyframes, while differential feature vector extraction is performed for non-keyframes in the sub-segment), obtaining feature subsets corresponding to the sub-segments. Feature subsets belonging to the same lens segment are integrated to obtain a scene feature set. The scene feature sets corresponding to each lens segment are integrated to obtain the video feature set corresponding to the IPTV video file. This solution significantly reduces the number of frames requiring full feature extraction through the layered processing logic of "lens decomposition → keyframe extraction → sub-segment feature extraction," focusing feature extraction on keyframes and their associated sub-segments, significantly reducing the amount of computation. This feature extraction solution strictly synchronizes features with video frames, ensuring that the features of each video frame can be derived by associating them with the key frames of its sub-segment, meeting the requirements of the end-edge-cloud collaborative architecture (i.e., the IPTV architecture of end-side nodes, edge nodes, and cloud nodes) for frame-level synchronized feature streams.

[0008] In combination with the first aspect, in a first possible implementation of the first aspect, the original frame sequence is lens-decomposed to determine multiple lens segments, including: downsampling the original frame sequence to obtain a preprocessed frame sequence, and performing color space conversion on the preprocessed frame sequence to obtain a corresponding Y channel frame sequence and UV channel frame sequence; performing brightness histogram statistics on the frame images in the Y channel frame sequence, and performing chromaticity histogram statistics on the frame images in the UV channel frame sequence; based on the brightness histogram statistics results of the i-th frame image and the (i-1)-th frame image, calculating the brightness distribution difference corresponding to the i-th frame image, and based on the chromaticity histogram statistics of the i-th frame image and the (i-1)-th frame image. Based on the histogram statistics, the chromaticity distribution difference corresponding to the i-th frame image is calculated, where 1<i≤n, and n is the total number of images in the original frame sequence. Optical flow analysis is performed on the i-th frame image and the (i-1)-th frame image in the Y channel frame sequence to obtain the pixel motion index corresponding to the i-th frame image, where the pixel motion index includes the average motion amplitude, motion direction consistency, and motion vector variance. Based on the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to the i-th frame image, it is determined whether the i-th frame image is a shot segmentation node, and all shot segmentation nodes are determined accordingly. Based on the shot segmentation nodes, the original frame sequence is shot-decomposed to determine multiple shot fragments.

[0009] Beneficial Effects: During shot decomposition, three key metrics—luminance histogram (calculating the difference in luminance distribution between two adjacent frames), chrominance histogram (calculating the difference in chrominance distribution between two adjacent frames), and optical flow analysis (motion amplitude, directional consistency, and vector variance)—are comprehensively used to determine shot segmentation nodes. Multimodal fusion enables more accurate identification of shot cuts: luminance and chrominance are primarily used to capture sudden changes in the overall color distribution of the image (such as scene changes), while optical flow analysis primarily detects dramatic changes in local motion (such as rapid transitions or object movement). This approach improves shot decomposition accuracy, reduces false positives, and accommodates both sudden (hard cuts) and gradual (fades). Taking into account the required precision for lens segmentation, the original frame sequence is downsampled (spatial downsampling, for example, reducing 1080P to 540P or even 360P) to generate a preprocessed frame sequence. This can effectively reduce the computational complexity of subsequent histogram statistics and optical flow analysis while meeting the conditions for lens segmentation analysis (with minimal loss in lens segmentation accuracy and a reduction of computing resource consumption by over 70%, resulting in an extremely high cost-effectiveness).

[0010] In combination with the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, determining whether the i-th frame image is a shot segmentation node based on the luminance distribution difference, chrominance distribution difference, and pixel motion index corresponding to the i-th frame image, and determining all shot segmentation nodes accordingly include: marking the i-th frame image as a candidate abrupt shot segmentation node if the luminance distribution difference corresponding to the i-th frame image is higher than a set luminance distribution difference, the chrominance distribution difference corresponding to the i-th frame image is higher than a set chrominance distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than a first set value; determining the x-th frame image as a candidate gradual shot segmentation node interval if the motion vector variance corresponding to x consecutive frames of images is higher than a second set value, and the calculated cumulative difference of the luminance distribution difference and the chrominance distribution difference corresponding to the x consecutive frames of images is higher than the set cumulative difference, and determining the x consecutive frames of images as a candidate gradual shot segmentation node interval, and determining candidate gradual shot segmentation nodes therefrom; and integrating the candidate abrupt shot segmentation nodes and the candidate gradual shot segmentation nodes, eliminating candidate shot segmentation nodes that are separated from adjacent candidate shot segmentation nodes by less than a set number of frames, to obtain all shot segmentation nodes.

[0011] Beneficial effect: According to different shot switching types, several shot segmentation node evaluation rules are mainly designed: if the brightness distribution difference corresponding to the i-th frame image is higher than the set brightness distribution difference, the chromaticity distribution difference corresponding to the i-th frame image is higher than the set chromaticity distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than the first set value, the i-th frame image is marked as a candidate mutation shot segmentation node (suitable for fast and accurate identification of mutation shots, through the combination of brightness / chromaticity distribution difference threshold and motion direction consistency index, to capture instantaneous mutations in picture color and structure, such as hard cutting, black field switching, etc., to avoid single detection logic from missing gradual scenes. At the same time, low motion consistency indicates that the main body of the picture is moving chaotically, and false mutation signals, such as flash interference, can be filtered out accordingly); if the motion vector variance corresponding to x consecutive frames (for example, 5 frames, 10 frames, etc.) is greater than the second set value, and the brightness distribution difference and chromaticity distribution difference corresponding to these x consecutive frames of images are ... consecutive frames of images are greater than the second set value, and the brightness distribution difference and chromaticity distribution difference corresponding to these consecutive frames of images are greater than the second set value, and the brightness distribution difference and chromaticity distribution difference corresponding to these consecutive frames of images are greater than the second set value, and the brightness distribution difference and chromaticity distribution If the calculated cumulative difference is higher than the set cumulative difference, the continuous x frames of image are determined to be candidate gradual shot segmentation node intervals, and candidate gradual shot segmentation nodes are determined therefrom (suitable for accurate identification of gradual shots, by using the motion vector variance and cumulative color difference of continuous frames, such as fade-in and fade-out and dissolve special effects, to identify slowly transitioning shot boundaries, while high variance indicates complex motion in the picture, such as rapid translation or rotation, which, in conjunction with the cumulative color difference, can enhance the reliability of gradual segmentation). Then, the candidate sudden shot segmentation nodes and the candidate gradual shot segmentation nodes are integrated, and candidate shot segmentation nodes whose intervals with adjacent candidate shot segmentation nodes are less than the set number of frames (the set number of frames here is converted from time to frame number based on time. For example, the shortest shot of a general IPTV program will not be less than 2 seconds in length. If the frame rate is 30, this x is set to 60 frames, while if the frame rate is 60, this x needs to be set to 120) are eliminated to obtain all shot segmentation nodes. This allows for efficient and accurate segmentation of IPTV video programs at the shot level, while eliminating redundant shot segmentation nodes whose intervals with adjacent nodes are less than the set number of frames. This can effectively eliminate brief jitters within the same shot that are not real switches, thus avoiding over-segmentation.

[0012] In combination with the first possible implementation method of the first aspect, in a third possible implementation method of the first aspect, adaptive key frame extraction is performed on each shot segment, including: for each shot segment: generating a sliding window of a set size, wherein the set size is less than a set number of frames; for the first sliding window of the current shot segment, taking the start frame of the shot segment as the starting point: determining a number of key frames by taking a set inter-frame interval; for non-first and last sliding windows of the current shot segment, taking the previous key frame as the starting point: determining a number of key frames in the sliding window based on the brightness distribution difference, chromaticity distribution difference and pixel motion index corresponding to each frame image in the current sliding window; for the last sliding window of the current shot segment, taking the end frame of the shot segment as the end point: determining a number of key frames by moving the set inter-frame interval backward from the previous key frame.

[0013] Beneficial Effects: Due to the diverse shot segments of IPTV video programs, fixed keyframe extraction strategies struggle to adapt to diverse dynamic shot scenarios, making it difficult to select appropriate keyframes for analysis. Consequently, this approach employs a sliding window analysis scheme. For each shot segment, sliding window analysis (sliding window sizes, such as 30 or 60 frames) is employed. Keyframes are determined by setting inter-frame intervals for the first and last windows. This approach can address both abrupt and gradual shot cuts, ensuring coverage of the initial and final stages of the shot segment and avoiding omissions of content from the beginning and end. For the intermediate window of a shot segment (i.e., the non-first and last sliding window), the previous keyframe is used as the starting point. Keyframes within the sliding window are determined based on the brightness and chromaticity distribution differences and pixel motion indicators corresponding to each frame within the current sliding window. In the analysis of non-head-and-tail sliding windows, fixed-step sliding window analysis may lead to a large amount of redundant calculations, which is not conducive to accurately determining the appropriate keyframes. (The selection of keyframes in this scheme is crucial because each subsequent keyframe will serve as the core of a sub-segment for full feature extraction, and other non-keyframes in the sub-segment with this keyframe as the core will use this keyframe as a benchmark for differential feature vector extraction. Therefore, the determination of keyframes will have a significant impact on the subsequent feature extraction process, and the extracted features will serve as the basis for auxiliary services for feature analysis of edge nodes.) Therefore, in the analysis of non-head-and-tail sliding windows, this scheme designs a dynamic sliding step and a comprehensive keyframe selection strategy. Utilizing the brightness distribution differences, chromaticity distribution differences, and pixel motion indicators calculated in the shot disassembly, appropriate evaluation schemes are designed according to different shot segment types (for example, some tend to be static shot segments, some are simple dynamic shot segments, and some are complex dynamic shots, etc.) as the basis for keyframe selection, thereby improving the accuracy of keyframe selection.

[0014] In combination with the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, based on the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to each frame of image in the current sliding window, a number of key frames in the sliding window are determined, including: calculating a motion state score corresponding to each frame of image based on the pixel motion index corresponding to each frame of image in the current sliding window, and calculating a color difference score corresponding to each frame of image based on the brightness distribution difference and chromaticity distribution difference corresponding to each frame of image; if there is an image in the current sliding window whose motion state score exceeds a motion state threshold, determining a key frames based on the motion state score and color difference score corresponding to each frame of image; if there is no image in the current sliding window whose motion state score exceeds the motion state threshold, but there is an image whose color difference score exceeds the color difference threshold, determining b key frames based on the motion state score and color difference score corresponding to each frame of image; if there is no image in the current sliding window whose motion state score exceeds the motion state threshold and no image whose color difference score exceeds the color difference threshold, determining c key frames based on the motion state score and color difference score corresponding to each frame of image, where a>b>c>1.

[0015] Beneficial effects: For different scenes (shot clips), this solution designs an evaluation mechanism for motion state scoring and color difference scoring. The motion state scoring reflects the dynamic complexity of the picture by quantifying the average motion amplitude, directional consistency, and motion vector variance (for example, in fast-action scenes, the motion state score is significantly higher). If there is an image with a motion state score exceeding the motion state threshold in the current sliding window, a key frame is determined based on the motion state score and color difference score corresponding to each frame (determined according to the size of the sliding window, generally a window size of 30 frames corresponds to 5 key frames); and the color difference score is calculated by combining the brightness and chromaticity differences, which is conducive to capturing sudden color changes in the picture (such as changes in light and shadow). If If there are no images with motion scores exceeding the motion threshold in the previous sliding window, but there are images with color difference scores exceeding the color difference threshold, b keyframes are determined based on the motion and color difference scores for each frame (for a 30-frame window, the corresponding number of keyframes, b, is 3). If there are no images with motion scores exceeding the motion threshold and no images with color difference scores exceeding the color difference threshold in the current sliding window, indicating a relatively regular, static-oriented shot, c keyframes are determined based on the motion and color difference scores for each frame (for a 30-frame window, the corresponding number of keyframes, c, is 2), where a > b > c > 1. This scheme allows for dense sampling within sliding windows with intense motion or color changes (such as explosions and transitions) and sparse sampling in static areas (such as fixed shots), achieving on-demand resource allocation. The resulting keyframes are more accurate and facilitate high-quality feature extraction. Furthermore, the preset values ​​of parameters a, b, and c enforce a maximum number of keyframes for different scenarios, preventing an uncontrolled increase in keyframe sampling density.

[0016] In conjunction with the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, calculating the motion state score corresponding to each frame of image based on the pixel motion index corresponding to each frame of image in the current sliding window includes: calculating the motion state score corresponding to the j-th frame of image in the current sliding window using the following formula: , in, Score the motion state corresponding to the j-th frame image in the current sliding window, 、 and is the weight parameter, is the average motion amplitude corresponding to the j-th frame image in the current sliding window, is the reference value of the average motion amplitude, is the consistency of the motion direction corresponding to the j-th frame image in the current sliding window, is the motion vector variance corresponding to the j-th frame image in the current sliding window, is the motion vector variance reference value; based on the brightness distribution difference and chromaticity distribution difference corresponding to each frame image, the color difference score corresponding to each frame image is calculated, including: using the following formula to calculate the color difference score corresponding to the j-th frame image in the current sliding window: , in, is the color difference score corresponding to the j-th frame image in the current sliding window, is the brightness distribution difference corresponding to the j-th frame image in the current sliding window, is the chromaticity distribution difference corresponding to the j-th frame image in the current sliding window, is the synergistic enhancement coefficient.

[0017] Beneficial Effects: The motion score calculation scheme uses logarithmic compression to reduce dimensional differences in motion amplitude (e.g., high-speed motion scenes have extremely large amplitudes), preventing this metric from dominating the motion score. High motion consistency is converted to a low contribution value, highlighting indicators of directional confusion (e.g., rapid camera shake or object collisions). Motion vector variance is approximately normalized (not standardized, but converted using a reference value). Weight coefficients are assigned to each factor (either empirically set or learnable to accommodate different scenarios, but given the complexity and increased computational effort involved, this scheme uses empirically set parameters as an example and considers different IPTV program types, assigning different weights). This allows for flexible adaptation to diverse video content and improves generalization across diverse IPTV programs. Reference values ​​are set based on empirical analysis, though global statistics can also be calculated in real time. The calculation scheme for color difference scores is based on the fact that the differences in brightness and chromaticity distribution in different shot scenes are not simply linear, but rather have a certain degree of synergy. When both the brightness and chromaticity distribution differences are high, the overall score should be significantly improved (to adapt to large changes in light, shadow, and color within the shot). In the case of a single factor (one high brightness and one low chromaticity distribution), the higher factor should dominate. For stable scenes, the score should be low, making them unsuitable for keyframes. Therefore, this color difference score calculation scheme is designed to more accurately achieve color difference scoring for each frame, and the synergy enhancement coefficient can be used as a means to balance sensitivity to different types of shot clips, improving adaptability to different types of shot clips.

[0018] In combination with the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, determining a key frames based on the motion state score and color difference score corresponding to each frame of image includes: calculating the comprehensive score corresponding to the j-th frame of image in the current sliding window using the following formula: , in, is the comprehensive score corresponding to the j-th frame image in the current sliding window, is the motion state threshold; based on the comprehensive score corresponding to each frame image in the current sliding window, determine the a key frames with the highest comprehensive scores.

[0019] Beneficial effect: For high-motion footage, the motion state score Manifested as exceeding the motion state threshold , As it gradually approaches 1, the motion state score accounts for a higher proportion in the comprehensive score, which can strengthen the selection of motion-dominated key frames.

[0020] In combination with the fifth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, determining b key frames based on the motion state score and color difference score corresponding to each frame of image includes: calculating the comprehensive score corresponding to the j-th frame of image in the current sliding window using the following formula: , in, is the comprehensive score corresponding to the j-th frame image in the current sliding window, is the color difference threshold; based on the comprehensive score corresponding to each frame image in the current sliding window, determine the b key frames with the highest comprehensive scores.

[0021] Beneficial effect: For low-motion footage, the motion score Do not exceed the exercise state threshold , at this time, it is necessary to perform a comprehensive calculation based on the color difference score and the motion status score. As the coefficient ( is not zero), so that the color difference weight can be adaptively enhanced according to the color difference score. In footage with smooth motion, key frames can still be triggered by color differences, thereby determining more appropriate key frames.

[0022] In combination with the fifth possible implementation manner of the first aspect, in an eighth possible implementation manner of the first aspect, determining c key frames based on the motion state score and color difference score corresponding to each frame of image includes: calculating the comprehensive score corresponding to the j-th frame of image in the current sliding window using the following formula: , in, is the comprehensive score corresponding to the j-th frame image in the current sliding window; based on the comprehensive score corresponding to each frame image in the current sliding window, determine the c key frames with the highest comprehensive scores.

[0023] Beneficial effect: For situations where both the motion status score and the color difference score are low, the two can be directly summed with equal weights (of course, other proportions can also be used). Generally, they will not be selected as key frames. However, if this is the case for all scenes in a sliding window, the one with the highest comprehensive score will be selected as the key frame (in this case, because the picture changes little, the number of key frames determined is relatively small. Compared with other scenes with larger changes, in this case, a key frame can dominate more non-key frames and serve as the "backbone" of more non-key frames thereafter).

[0024] In combination with the fifth possible implementation method of the first aspect, in the ninth possible implementation method of the first aspect, a sub-segment is determined with the key frame as the core, and then the image features of each frame image in the sub-segment are differentially extracted to obtain a feature subset corresponding to the sub-segment, including: dividing the current shot segment into several sub-segments based on the key frame, the first frame image of each sub-segment is a key frame, wherein each sub-segment does not contain a key frame other than the first frame image; for each sub-segment: using ResNet50 to extract the full feature vector of the key frame of the sub-segment, and, based on the key frame of this sub-segment, using MobileNetV3 to extract the differential feature vector of each non-key frame of the sub-segment, and then integrating the feature vectors based on the frame order of the sub-segment to obtain the feature subset corresponding to this sub-segment.

[0025] Beneficial effects: For each sub-segment, this solution extracts global features through ResNet50 full-frame processing and uses MobileNetV3 to extract differential features of non-key frames, which can effectively reduce the processing volume. MobileNetV3 can also be designed to be lightweight, further reducing computational overhead, making it suitable for large-scale cloud-based video processing.

[0026] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 Flowchart of the video program feature extraction method based on artificial intelligence provided in an embodiment of the present application.

[0029] Figure 2 Schematic diagram for determining shot segmentation nodes.

[0030] Figure 3 Schematic diagram for determining key frames.

[0031] Figure 4 Schematic diagram of the differential feature extraction process. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0033] See also Figure 1 , Figure 1 The flowchart of the video program feature extraction method based on artificial intelligence provided in the embodiment of the present application is applied to the cloud node, including step S10, step S20, step S30, step S40, and step S50.

[0034] To extract features from IPTV video files stored on cloud nodes and ensure that each frame has corresponding features, so as to adapt to the end-edge-cloud (end-side node-edge node-cloud node) IPTV architecture, this embodiment takes the feature extraction process of an IPTV video file as an example.

[0035] First, the cloud node may execute step S10.

[0036] Step S10: Read the IPTV video file stored in the cloud node and decode it into an original frame sequence.

[0037] In this embodiment, the cloud node can read the stored IPTV video file to be processed and decode it into a raw frame sequence, where the number of frames in the raw frame sequence is n. For example, a 40-minute TV series episode with a frame rate of 30 would contain 40*60*30 = 72,000 frames at a resolution of 1080p (i.e., 1920×1080 pixels). This is just an example; actual resolutions may be higher or lower, and video programs of other durations and frame rates are not limited.

[0038] After obtaining the original frame sequence, the cloud node may execute step S20.

[0039] Step S20: Decompose the original frame sequence into shots to determine a plurality of shot segments.

[0040] In this embodiment, the cloud node can downsample the original frame sequence (before this, the bottom area can be blocked or cropped, such as the bottom 1 / 5 area, to prevent subtitles from affecting the extraction of various indicators) (downsampling here refers to spatial downsampling, reducing the resolution, for example, to 540P, 360P, etc. (if the bottom area is blocked or cropped, the actual resolution may not be strictly 540P, but the size of each frame image is still the same, which does not affect the processing, so no additional explanation is given) to reduce the pixel count of a single frame image, greatly improving the processing speed while losing a small amount of shot splitting accuracy), to obtain a preprocessed frame sequence, the number of frames of the preprocessed frame sequence is n, the same as the number of frames of the original frame sequence.

[0041] Then, the preprocessed frame sequence is converted into a color space from RGB color space to YUV color space to obtain the corresponding Y channel frame sequence and UV channel frame sequence.

[0042] Based on this, we can perform luminance histogram statistics on the frames in the Y channel frame sequence, perform chrominance histogram statistics on the frames in the UV channel frame sequence, and perform optical flow analysis based on the i-th frame image and the (i-1)-th frame image in the Y channel frame sequence to obtain the pixel motion index corresponding to the i-th frame image. These three processes can be processed simultaneously to improve efficiency.

[0043] Exemplarily, the cloud node can perform luminance histogram statistics on the frame images in the Y channel frame sequence, and then calculate the luminance distribution difference corresponding to the i-th frame image (the absolute value of the luminance histogram statistics of the subsequent frame minus the luminance histogram statistics of the previous frame) based on the luminance histogram statistics of the i-th frame image and the (i-1)-th frame image, where 1<i≤n. For the first frame, the luminance histogram statistics can be used as the luminance distribution difference. In this embodiment, the first frame of the original frame sequence is to be used as a key frame. In addition, chrominance histogram statistics can be performed on the frame images in the UV channel frame sequence, and based on the chrominance histogram statistics of the i-th frame image and the (i-1)-th frame image, the chrominance distribution difference corresponding to the i-th frame image (the absolute value of the chrominance histogram statistics of the subsequent frame minus the chrominance histogram statistics of the previous frame) can be calculated, where 1<i≤n. Similarly, for the first frame, the chrominance histogram statistics can be used as the chrominance distribution difference. It should be noted that the brightness distribution difference corresponding to the i-th frame image here is calculated using the frame images in the Y channel frame sequence and also corresponds to the i-th frame image in the original frame sequence; and the chromaticity distribution difference corresponding to the i-th frame image is calculated using the frame images in the UV channel frame sequence and also corresponds to the i-th frame image in the original frame sequence.

[0044] Optical flow analysis takes the Y channel frame sequence as input and uses the optical flow analysis to output the corresponding parameters. The parameters here refer to the pixel motion indicators corresponding to the i-th frame image (1<i≤n), including the average motion amplitude, motion direction consistency and motion vector variance. For the first frame, the initial value is set to: average motion amplitude , consistency of movement direction and motion vector variance It should also be noted that the pixel motion index corresponding to the i-th frame image here is calculated based on the frame images in the Y channel frame sequence, and also corresponds to the i-th frame image in the original frame sequence.

[0045] The average motion amplitude corresponding to the i-th frame image: , (1) in, is the average motion amplitude corresponding to the i-th frame image, is the total number of feature points of the i-th frame image in the Y channel frame sequence, The first feature points (the optical flow method can be a dense optical flow method, that is, each pixel of the frame image is used as a feature point, or the optical flow method can be a sparse optical flow method, that is, a fixed portion of pixel points extracted from the frame image is used as a feature point, or the comprehensive pixel points in an area are calculated as a feature point and extracted from the frame image. area, thus obtaining feature points (not limited here) The first The motion vector of each feature point is calculated by the optical flow method. for The mold length.

[0046] The consistency of motion direction corresponding to the i-th frame image: , (2) in, is the consistency of the motion direction corresponding to the i-th frame image, For the The direction angle of the motion vector of the feature point, The main direction angle is obtained by taking the average direction angle of the motion vectors of all feature points in the frame image.

[0047] The motion vector variance corresponding to the i-th frame image: , (3) in, is the motion vector variance corresponding to the i-th frame image.

[0048] After obtaining the brightness distribution difference, chromaticity distribution difference and pixel motion index corresponding to each frame image, the cloud node can determine whether the i-th frame image is a shot segmentation node based on the brightness distribution difference, chromaticity distribution difference and pixel motion index corresponding to the i-th frame image, and thus determine all shot segmentation nodes.

[0049] During the shot decomposition process, three key metrics—luminance histogram (calculating the difference in luminance distribution between two adjacent frames), chrominance histogram (calculating the difference in chrominance distribution between two adjacent frames), and optical flow analysis (motion amplitude, directional consistency, and vector variance)—are comprehensively used to determine shot segmentation nodes. Multimodal fusion enables more accurate identification of shot cuts: luminance and chrominance are primarily used to capture sudden changes in the overall color distribution of the image (such as scene changes), while optical flow analysis primarily detects dramatic changes in local motion (such as rapid transitions or object movement). This approach improves shot decomposition accuracy, reduces false positives, and accommodates both sudden (such as hard cuts) and gradual (such as fades). Taking into account the required precision for shot segmentation, the original frame sequence is downsampled (spatial downsampling, for example, reducing 1080P to 540P or even 360P. Different downsampling indicators can be determined for different types of IPTV video programs. For example, action-type sports programs are reduced to 540P, and urban soap operas are reduced to 360P) to generate a preprocessed frame sequence. This can effectively reduce the computational complexity of subsequent histogram statistics and optical flow analysis while meeting the conditions for shot segmentation analysis (with minimal loss in shot segmentation accuracy and a reduction of computing resource consumption by over 70%, resulting in an extremely high cost-effectiveness).

[0050] In this embodiment, based on different shot switching types (hard cut and gradual transition), the following shot segmentation node evaluation rules are designed: If the brightness distribution difference corresponding to the i-th frame image is higher than the set brightness distribution difference, the chromaticity distribution difference corresponding to the i-th frame image is higher than the set chromaticity distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than the first set value, the cloud node can mark the i-th frame image as a candidate sudden shot segmentation node.

[0051] This solution is suitable for the rapid and accurate identification of sudden changes in shot quality (i.e., hard cuts). By combining the brightness / chromaticity distribution difference threshold with the motion direction consistency index, it can capture instantaneous sudden changes in image color and structure, such as hard cuts and black field switching, to avoid missing gradual changes due to a single detection logic. At the same time, low motion consistency indicates that the main body of the image is moving chaotically, which can be used to filter out false sudden changes, such as flash interference.

[0052] If the motion vector variance corresponding to x consecutive image frames (e.g., 5 frames, 10 frames, etc.) is greater than a second set value, and the calculated cumulative difference of the brightness distribution difference and the chromaticity distribution difference corresponding to these x consecutive image frames is greater than the set cumulative difference, these x consecutive image frames are determined to be candidate gradient shot segmentation node intervals, and a candidate gradient shot segmentation node is determined therefrom.

[0053] This solution is suitable for accurately identifying gradual transition shots. It uses the motion vector variance and cumulative color differences of consecutive frames, such as fade-in, fade-out, and dissolve effects, to identify slowly transitioning shot boundaries. High variance indicates complex motion in the image, such as rapid translation or rotation. Combined with the cumulative color difference, it can enhance the reliability of gradual transition segmentation.

[0054] After that, the cloud node can integrate the candidate sudden shot segmentation nodes and the candidate gradual shot segmentation nodes, remove the candidate shot segmentation nodes whose interval with the adjacent candidate shot segmentation nodes is less than the set number of frames, and obtain all the shot segmentation nodes, such as Figure 2 As shown, a small number of determined shot segmentation nodes are schematically shown.

[0055] The frame rate setting here is based on time, converted to frame count. For example, the shortest shot of a typical IPTV program is no less than 2 seconds long. If the frame rate is 30, x is set to 60 frames, while if the frame rate is 60, x needs to be set to 120. This allows for efficient and accurate segmentation of IPTV video programs at the shot level, while eliminating redundant shot segmentation nodes that are less than the set number of frames away from adjacent nodes. This effectively eliminates brief jitter within the same shot that is not a true cut, and avoids over-segmentation.

[0056] Based on this, the cloud node can perform shot decomposition on the original frame sequence based on the shot segmentation node to determine multiple shot segments. Note that the shot decomposition here is performed on the original frame sequence, not the pre-processed frame sequence, nor the Y channel frame sequence or UV channel frame sequence.

[0057] After determining the shot segment, the cloud node may execute step S30.

[0058] Step S30: adaptively extract key frames from each shot segment, determine sub-segments with the key frames as the core, and then differentially extract image features of each frame image in the sub-segment to obtain feature subsets corresponding to the sub-segment.

[0059] Since IPTV video programs have a wide variety of shot segments, fixed keyframe extraction strategies are difficult to adapt to a variety of dynamic shot scenes, making it difficult to select appropriate keyframes for analysis. Therefore, this solution designs a sliding window analysis solution. In this embodiment, for each shot segment: The cloud node may generate a sliding window of a set size, wherein the set size is smaller than the set number of frames. For example, the size of the sliding window is designed to be 30 frames (or 60 frames, or other sizes).

[0060] For the first sliding window of the current shot clip (for example, a 300-frame shot clip), starting from the shot clip's start frame, a set inter-frame interval (for example, 5 frames) is used to determine a number of key frames (the key frames are frames 1, 7, 13, 19, and 25). For the last sliding window of the current shot clip (the number of frames from the previous key frame to the end frame of this shot clip is exactly 30 frames or less), starting from the shot clip's end frame, a set inter-frame interval (also 5 frames) is used to determine a number of key frames.

[0061] The first and last windows use a scheme of setting the interval between frames to determine the key frames. This scheme can cope with both sudden and gradual lens switching dominated shot segments, ensuring coverage of the starting and ending stages of the shot segments and avoiding omission of the beginning and end content.

[0062] For non-first and last sliding windows of the current shot segment, the cloud node needs to use the previous key frame as the starting point: based on the brightness distribution difference, chromaticity distribution difference and pixel motion index corresponding to each frame image in the current sliding window, determine several key frames in the sliding window.

[0063] In the analysis of non-head-and-tail sliding windows, sliding window analysis with a fixed step size may result in a large amount of redundant calculations, which is not conducive to accurately determining the appropriate keyframes (in this embodiment, the selection of keyframes is crucial because each subsequent keyframe will serve as the core of a sub-segment for full feature extraction, and other non-keyframes in the sub-segment with the keyframe as the core will use the keyframe as a benchmark for differential feature vector extraction. Therefore, the determination of keyframes will have a significant impact on the subsequent feature extraction process, and the extracted features will serve as the basis for providing auxiliary services for feature analysis of edge nodes). Therefore, in the analysis of non-head-and-tail sliding windows, this solution designs a dynamic sliding step size and a comprehensive keyframe selection strategy. Utilizing the brightness distribution differences, chromaticity distribution differences, and pixel motion indicators calculated in the shot disassembly, appropriate evaluation schemes are designed according to different shot segment types (for example, some tend to be static shot segments, some are simple dynamic shot segments, and some are complex dynamic shots, etc.) as the basis for keyframe selection, thereby improving the accuracy of keyframe selection.

[0064] Exemplarily, the cloud node may calculate a motion state score corresponding to each frame of image based on a pixel motion index corresponding to each frame of image in the current sliding window.

[0065] Specifically, the cloud node uses the following formula to calculate the motion state score corresponding to the j-th frame image in the current sliding window: , (4) in, Score the motion state of the j-th frame image (corresponding to a unique original frame number) in the current sliding window. 、 and is the weight parameter, is the average motion amplitude corresponding to the j-th frame image in the current sliding window, is the reference value of the average motion amplitude, is the consistency of the motion direction corresponding to the j-th frame image in the current sliding window, is the motion vector variance corresponding to the j-th frame image in the current sliding window, is the motion vector variance reference value.

[0066] The motion score calculation scheme uses logarithmic compression to reduce dimensional differences in motion amplitude (e.g., high-speed motion scenes have extremely large amplitudes) to prevent this metric from dominating the motion score. High motion consistency is converted to a low contribution value, highlighting indicators of directional confusion (e.g., rapid camera shake or object collisions). Motion vector variance is approximately normalized (not standardized, but converted using a reference value). Weight coefficients are assigned to each factor (either empirically set or learnable to accommodate different scenarios, but given the complexity and increased computational effort involved, this scheme uses empirically set parameters as an example, taking into account different IPTV program types and assigning different weights). This allows for flexible adaptation to diverse video content and improves generalization across diverse IPTV programs. Reference values ​​are set based on empirical analysis, though global statistics can also be calculated in real time.

[0067] Furthermore, the cloud node can calculate the color difference score corresponding to each frame of image based on the brightness distribution difference and chromaticity distribution difference corresponding to each frame of image.

[0068] Specifically, the color difference score corresponding to the j-th frame image in the current sliding window is calculated using the following formula: , (5) in, is the color difference score corresponding to the j-th frame image in the current sliding window, is the brightness distribution difference corresponding to the j-th frame image in the current sliding window, is the chromaticity distribution difference corresponding to the j-th frame image in the current sliding window, is the synergistic enhancement coefficient.

[0069] In the color difference scoring calculation scheme, since the luminance distribution difference and chrominance distribution difference in several different shot scenes are not simply linearly related, but rather have a certain degree of synergy, when the luminance distribution difference and chrominance distribution difference are both high, the overall score should be significantly improved (to adapt to the large changes in light, shadow, and color within the shot segment); while in the case of a single factor (one high luminance distribution difference and one low chrominance distribution difference), the higher factor should dominate; for stable scenes, the score should be low and unsuitable as a keyframe. Therefore, this color difference scoring calculation scheme is designed to more accurately achieve color difference scoring for each frame, and the synergy enhancement coefficient can be used as a means to balance sensitivity to different types of shot segments, improving adaptability to different types of shot segments.

[0070] After calculating the motion state score and color difference score for each frame in the current sliding window, the cloud node can make judgments and take corresponding actions: If there are images with motion state scores exceeding the motion state threshold in the current sliding window (to improve reliability, the judgment conditions can be set more strictly, for example, there are at least three frames of images with motion state scores exceeding the motion state threshold in the current sliding window), a key frames are determined based on the motion state score and color difference score corresponding to each frame of the image.

[0071] Specifically, the cloud node can use the following formula to calculate the comprehensive score corresponding to the j-th frame image in the current sliding window: , (6) in, is the comprehensive score corresponding to the j-th frame image in the current sliding window, is the motion state threshold (for sports events, 0.7 is used as an example, for film and television programs, 0.5 is used as an example). For high-motion footage, the motion state score Manifested as exceeding the motion state threshold , As it gradually approaches 1, the motion state score accounts for a higher proportion in the comprehensive score, which can strengthen the selection of motion-dominated key frames.

[0072] Based on this, the cloud node can determine the a key frames with the highest comprehensive scores based on the comprehensive scores corresponding to each frame image in the current sliding window (corresponding to a sliding window of 30 frames, the corresponding number of key frames a is 5 for example).

[0073] If there are no images with motion state scores exceeding the motion state threshold in the current sliding window (correspondingly, in the reliability judgment condition, it is designed that: there are less than 3 frames of images with motion state scores exceeding the motion state threshold in the current sliding window), but there are images with color difference scores exceeding the color difference threshold (also designed that there are at least 3 frames of images with color difference scores exceeding the color difference threshold in the current sliding window), then the cloud node can determine b key frames based on the motion state score and color difference score corresponding to each frame of the image.

[0074] Specifically, the cloud node can use the following formula to calculate the comprehensive score corresponding to the j-th frame image in the current sliding window: , (7) in, is the comprehensive score corresponding to the j-th frame image in the current sliding window, is the color difference threshold ( It is not zero, and is generally between 0.3 and 0.4. Special scenes such as fireworks shows and light shows need to increase the value, and it is expected to be no less than 0.5. For low-motion footage, the motion state score Do not exceed the exercise state threshold , at this time, it is necessary to perform a comprehensive calculation based on the color difference score and the motion status score. As a coefficient, the color difference weight can be adaptively enhanced according to the color difference score. In footage with smooth motion, key frames can still be triggered by color differences, thereby determining more appropriate key frames.

[0075] Accordingly, the cloud node can determine b key frames with the highest comprehensive scores (in this embodiment, b is 3) based on the comprehensive scores corresponding to each frame image in the current sliding window.

[0076] However, if there are no images in the current sliding window whose motion state score exceeds the motion state threshold (correspondingly, in the reliability judgment condition, it is designed as: there are less than 3 frames of images with motion state scores exceeding the motion state threshold in the current sliding window), and there are no images with color difference scores exceeding the color difference threshold (there are less than 3 frames of images with color difference scores exceeding the color difference threshold in the current sliding window), then the cloud node can determine c key frames based on the motion state score and color difference score corresponding to each frame of the image, where a>b>c>1.

[0077] Specifically, the cloud node can use the following formula to calculate the comprehensive score corresponding to the j-th frame image in the current sliding window: , (8) in, is the comprehensive score corresponding to the jth frame in the current sliding window. For low motion scores and low color difference scores, the two scores are simply weighted and summed (other weightings are possible). Generally, these frames will not be selected as keyframes. However, if this is the case for all frames in a sliding window, the frame with the highest comprehensive score will be selected as the keyframe. (In this case, because the image changes slightly, fewer keyframes are determined. Compared to other scenes with larger changes, a single keyframe can dominate more non-keyframes, serving as the backbone for subsequent non-keyframes.)

[0078] Accordingly, the cloud node can determine c key frames with the highest comprehensive scores (c is 2 in this embodiment) based on the comprehensive scores corresponding to each frame image in the current sliding window.

[0079] For different scenes (shot clips), this embodiment designs an evaluation mechanism for motion state scoring and color difference scoring. The motion state scoring reflects the dynamic complexity of the picture by quantifying the average motion amplitude, directional consistency, and motion vector variance (for example, the motion state score is significantly higher in fast action scenes). If there is an image with a motion state score exceeding the motion state threshold in the current sliding window, a key frame is determined based on the motion state score and color difference score corresponding to each frame (determined by the size of the sliding window, generally a key frame number a is 5 for a window size of 30 frames); and the color difference score is calculated by combining the brightness and chromaticity differences, which is conducive to capturing sudden color changes in the picture (such as changes in light and shadow). If the current sliding window has an image with a motion state score exceeding the motion state threshold, a key frame is determined based on the motion state score and color difference score corresponding to each frame (determined by the size of the sliding window, generally a key frame number a is 5 for a window size of 30 frames); and the color difference score is calculated by combining the brightness and chromaticity differences, which is conducive to capturing sudden color changes in the picture (such as changes in light and shadow). If there are no images with motion scores exceeding the motion threshold within the sliding window, but there are images with color difference scores exceeding the color difference threshold, b keyframes are determined based on the motion and color difference scores for each frame (for a 30-frame window, the corresponding number of keyframes, b, is 3). If there are no images with motion scores exceeding the motion threshold and no images with color difference scores exceeding the color difference threshold within the current sliding window, indicating a relatively regular, more static shot, c keyframes are determined based on the motion and color difference scores for each frame (for a 30-frame window, the corresponding number of keyframes, c, is 2), where a > b > c > 1. This scheme allows for dense sampling within the sliding window of areas with intense motion or color changes (such as explosions and transitions) and sparse sampling in static areas (such as fixed shots), achieving on-demand resource allocation. The resulting keyframes are more accurate and facilitate high-quality feature extraction. Furthermore, the preset values ​​of the parameters a, b, and c enforce a maximum number of keyframes for different scenarios, preventing an uncontrolled increase in keyframe sampling density.

[0080] After determining the key frame, the cloud node can deduplicate the key frame and determine the sub-segment with the key frame as the core: specifically, the current shot segment is divided into several sub-segments with the key frame as the node, the first frame image of each sub-segment is the key frame, and each sub-segment does not contain key frames other than the first frame image, such as Figure 3 shown.

[0081] After determining the sub-segment with the key frame as the core, the cloud node can differentially extract the image features of each frame image in the sub-segment to obtain the feature subset corresponding to the sub-segment.

[0082] Exemplarily, for each sub-segment: the cloud node can use ResNet50 to extract the full feature vector of the key frame of the sub-segment (for example, the extracted full feature vector dimension is 1024 dimensions, corresponding to a frame image in the original frame sequence), and, based on the key frame of this sub-segment, use MobileNetV3 to extract the differential feature vector of each non-key frame of the sub-segment (for example, the extracted full feature vector dimension is 256 dimensions, which also corresponds to the image in the original frame sequence frame by frame), and then integrate the feature vectors based on the frame order of the sub-segment (integration can be performed in true order) to obtain the feature subset corresponding to this sub-segment.

[0083] The differential feature extraction process is detailed in Figure 4 The key frame (first frame) of this sub-segment is used as the reference frame. Optical flow is estimated for each non-key frame of this sub-segment to obtain an optical flow field (pixel-level motion information, i.e., the horizontal and vertical displacement of each pixel). This is used for motion-compensated alignment (i.e., alignment of non-key frames with key frames). The differential image between the non-key frames and the key frames is further obtained and input into MobileNetV3 (a lightweight version such as MobileNetV3-Small can be selected). The 256-dimensional differential feature vector of the non-key frame is extracted and output. It should be noted that the vector dimensions of the key frames are different from those of the non-key frames, which does not affect subsequent applications. Differential coding mode can be used for pre-encoding, eliminating the need for encoding during transmission, thereby improving the operational efficiency of the IPTV end-edge-cloud architecture.

[0084] This solution extracts global features through ResNet50 full-frame processing and uses MobileNetV3 to extract differential features from non-key frames, which can effectively reduce the processing volume. MobileNetV3 can also be designed to be lightweight, further reducing computational overhead, making it suitable for large-scale video processing in the cloud.

[0085] After obtaining the feature subset corresponding to each sub-segment, the cloud node may execute step S40.

[0086] Step S40: Integrate the feature subsets belonging to the same shot segment to obtain a scene feature set.

[0087] In this embodiment, the cloud node can integrate feature subsets belonging to the same shot segment according to the frame sequence to obtain a scene feature set (or shot feature set).

[0088] Afterwards, the cloud node may execute step S50.

[0089] Step S50: Integrate the scene feature set corresponding to each shot segment to obtain a video feature set corresponding to the IPTV video file.

[0090] In this embodiment, the cloud node can integrate the scene feature set corresponding to each shot segment according to the frame sequence to obtain the video feature set corresponding to the IPTV video file. The number of features in the video feature set is the same as the number of frames in the original frame sequence in the IPTV video file, and can correspond one to one.

[0091] In summary, the embodiments of the present application provide an artificial intelligence-based method for extracting video program features. This method reads IPTV video files stored on cloud nodes, decodes them into raw frame sequences, performs shot decomposition to determine multiple shot segments, then performs adaptive keyframe extraction on each shot segment, and determines sub-segments based on the key frames. The method then differentially extracts the image features of each frame in the sub-segment (full feature vector extraction is performed for key frames, while differential feature vector extraction is performed for non-key frames in the sub-segment), obtaining feature subsets corresponding to the sub-segments. Feature subsets belonging to the same shot segment are integrated to obtain a scene feature set. The scene feature sets corresponding to each shot segment are integrated to obtain a video feature set corresponding to the IPTV video file. This solution significantly reduces the number of frames requiring full feature extraction through the layered processing logic of "shot decomposition → keyframe extraction → sub-segment feature extraction," focusing feature extraction on key frames and their associated sub-segments, significantly reducing computational complexity. This feature extraction solution strictly synchronizes features with video frames, ensuring that the features of each video frame can be derived by associating them with the key frames of its sub-segment, meeting the requirements of the end-edge-cloud collaborative architecture (i.e., the IPTV architecture of end-side nodes, edge nodes, and cloud nodes) for frame-level synchronized feature streams.

[0092] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A video program feature extraction method based on artificial intelligence, characterized in that: Applied to cloud nodes, including: Read the IPTV video files stored in the cloud node and decode them into original frame sequences; Decompose the original frame sequence into shots to identify multiple shot segments; Adaptively extract keyframes from each shot segment, and determine subsegments based on the keyframes. Then, differentially extract the image features of each frame in the subsegment to obtain the feature subset corresponding to the subsegment. Integrate the feature subsets belonging to the same shot segment to obtain the scene feature set; The scene feature set corresponding to each shot segment is integrated to obtain the video feature set corresponding to the IPTV video file.

2. The method for extracting video program features based on artificial intelligence according to claim 1, characterized in that: The original frame sequence is disassembled to identify multiple shot segments, including: Downsampling the original frame sequence to obtain a preprocessed frame sequence, and performing color space conversion on the preprocessed frame sequence to obtain corresponding Y channel frame sequence and UV channel frame sequence; Perform brightness histogram statistics on the frame images in the Y channel frame sequence, and perform chromaticity histogram statistics on the frame images in the UV channel frame sequence; Based on the luminance histogram statistics of the i-th frame image and the (i-1)-th frame image, calculate the luminance distribution difference corresponding to the i-th frame image; based on the chrominance histogram statistics of the i-th frame image and the (i-1)-th frame image, calculate the chrominance distribution difference corresponding to the i-th frame image, 1<i≤n, n is the total number of images in the original frame sequence; Based on the optical flow analysis of the i-th frame image and the (i-1)-th frame image in the Y channel frame sequence, the pixel motion index corresponding to the i-th frame image is obtained, where the pixel motion index includes the average motion amplitude, motion direction consistency and motion vector variance; Based on the brightness distribution difference, chromaticity distribution difference and pixel motion index corresponding to the i-th frame image, determine whether the i-th frame image is a shot segmentation node, and determine all shot segmentation nodes accordingly; The original frame sequence is shot-decomposed based on the shot segmentation node to determine multiple shot segments.

3. The method for extracting video program features based on artificial intelligence according to claim 2, characterized in that: Based on the brightness distribution difference, chromaticity distribution difference and pixel motion index corresponding to the i-th frame image, determine whether the i-th frame image is a shot segmentation node, and thus determine all shot segmentation nodes, including: If the brightness distribution difference corresponding to the i-th frame image is higher than the set brightness distribution difference, the chromaticity distribution difference corresponding to the i-th frame image is higher than the set chromaticity distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than the first set value, mark the i-th frame image as a candidate sudden shot segmentation node; If the motion vector variance corresponding to the x consecutive image frames is greater than a second set value, and the calculated cumulative difference of the brightness distribution difference and the chromaticity distribution difference corresponding to the x consecutive image frames is greater than the set cumulative difference, the x consecutive image frames are determined to be a candidate gradient shot segmentation node interval, and a candidate gradient shot segmentation node is determined therefrom; The candidate sudden shot segmentation nodes and the candidate gradual shot segmentation nodes are integrated, and the candidate shot segmentation nodes whose intervals with adjacent candidate shot segmentation nodes are less than the set number of frames are eliminated to obtain all shot segmentation nodes.

4. The method for extracting video program features based on artificial intelligence according to claim 2, characterized in that: Adaptive keyframe extraction is performed on each shot segment, including: For each shot clip: generating a sliding window of a set size, wherein the set size is smaller than a set number of frames; For the first sliding window of the current shot segment, starting from the start frame of the shot segment, a number of key frames are determined using the set inter-frame intervals; For non-first and last sliding windows of the current shot segment, starting from the previous key frame, a number of key frames within the sliding window are determined based on the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to each frame image within the current sliding window; For the last sliding window of the current shot segment, the end frame of the shot segment is taken as the end point: a number of key frames are determined by moving the previous key frame backward by the interval between frames set.

5. The method for extracting video program features based on artificial intelligence according to claim 4, characterized in that: Based on the brightness distribution difference, chromaticity distribution difference and pixel motion index corresponding to each frame image in the current sliding window, several key frames in the sliding window are determined, including: Calculate the motion state score of each frame based on the pixel motion index of each frame in the current sliding window, and calculate the color difference score of each frame based on the brightness distribution difference and chromaticity distribution difference of each frame; If there is an image with a motion state score exceeding the motion state threshold in the current sliding window, a key frame is determined based on the motion state score and color difference score corresponding to each frame of the image; If there is no image with a motion state score exceeding the motion state threshold in the current sliding window, but there is an image with a color difference score exceeding the color difference threshold, b key frames are determined based on the motion state score and color difference score corresponding to each frame of the image; If there is no image with a motion state score exceeding the motion state threshold and no image with a color difference score exceeding the color difference threshold in the current sliding window, c key frames are determined based on the motion state score and color difference score corresponding to each frame of the image, where a>b>c>1.

6. The method for extracting video program features based on artificial intelligence according to claim 5, characterized in that: Based on the pixel motion index corresponding to each frame image in the current sliding window, the motion state score corresponding to each frame image is calculated, including: The following formula is used to calculate the motion state score corresponding to the j-th frame image in the current sliding window: ， in, Score the motion state corresponding to the j-th frame image in the current sliding window, 、 and is the weight parameter, is the average motion amplitude corresponding to the j-th frame image in the current sliding window, is the reference value of the average motion amplitude, is the consistency of the motion direction corresponding to the j-th frame image in the current sliding window, is the motion vector variance corresponding to the j-th frame image in the current sliding window, is the motion vector variance reference value; Based on the brightness distribution difference and chromaticity distribution difference corresponding to each frame of image, the color difference score corresponding to each frame of image is calculated, including: The following formula is used to calculate the color difference score corresponding to the j-th frame image in the current sliding window: ， in, is the color difference score corresponding to the j-th frame image in the current sliding window, is the brightness distribution difference corresponding to the j-th frame image in the current sliding window, is the chromaticity distribution difference corresponding to the j-th frame image in the current sliding window, is the synergistic enhancement coefficient.

7. The method for extracting video program features based on artificial intelligence according to claim 6, characterized in that: Based on the motion state score and color difference score corresponding to each frame image, a key frames are determined, including: The following formula is used to calculate the comprehensive score corresponding to the j-th frame image in the current sliding window: ， in, is the comprehensive score corresponding to the j-th frame image in the current sliding window, is the motion state threshold; Based on the comprehensive score corresponding to each frame image in the current sliding window, a key frames with the highest comprehensive scores are determined.

8. The method for extracting video program features based on artificial intelligence according to claim 6, characterized in that: Based on the motion state score and color difference score corresponding to each frame image, b key frames are determined, including: The following formula is used to calculate the comprehensive score corresponding to the j-th frame image in the current sliding window: ， in, is the comprehensive score corresponding to the j-th frame image in the current sliding window, is the color difference threshold; Based on the comprehensive score corresponding to each frame image in the current sliding window, b key frames with the highest comprehensive scores are determined.

9. The method for extracting video program features based on artificial intelligence according to claim 6, characterized in that: Based on the motion state score and color difference score corresponding to each frame image, c key frames are determined, including: The following formula is used to calculate the comprehensive score corresponding to the j-th frame image in the current sliding window: ， in, is the comprehensive score corresponding to the j-th frame image in the current sliding window; Based on the comprehensive score corresponding to each frame image in the current sliding window, the c key frames with the highest comprehensive scores are determined.

10. The method for extracting video program features based on artificial intelligence according to claim 6, characterized in that: The sub-segments are determined with the key frames as the core, and then the image features of each frame in the sub-segment are differentially extracted to obtain the feature subset corresponding to the sub-segment, including: Dividing the current shot segment into a plurality of sub-segments based on the key frames, wherein the first frame image of each sub-segment is a key frame, wherein each sub-segment does not contain a key frame other than the first frame image; For each sub-segment: ResNet50 is used to extract the full feature vectors of the key frames of the sub-segment, and based on the key frames of this sub-segment, MobileNetV3 is used to extract the differential feature vectors of each non-key frame of the sub-segment. The feature vectors are then integrated based on the frame order of the sub-segment to obtain the feature subset corresponding to this sub-segment.

Images