An artificial intelligence-based video program feature extraction method

Abstract

The application provides a video program feature extraction method based on artificial intelligence, relates to the technical field of IPTV, is applied to a cloud node, and comprises the following steps: reading an IPTV video file stored in the cloud node and decoding the IPTV video file into an original frame sequence; carrying out shot disassembly on the original frame sequence to determine a plurality of shot segments; carrying out adaptive key frame extraction on each shot segment, determining a subsegment with the key frame as the core, differentially extracting the image features of each frame of image in the subsegment, and obtaining a feature subset corresponding to the subsegment; integrating the feature subsets belonging to the same shot segment to obtain a scene feature set; and integrating the scene feature sets corresponding to each shot segment to obtain a video feature set corresponding to the IPTV video file. The scheme sets a new feature extraction scheme, adapts to a new IPTV architecture, and is beneficial to improving the extraction efficiency of the IPTV video program feature set.

Description

Technical Field

[0001] This application relates to the field of IPTV technology, and more specifically, to a method for extracting video program features based on artificial intelligence. Background Technology

[0002] In existing IPTV architectures, various ancillary services are typically provided in the cloud and embedded in video programs. This approach lacks flexibility and is difficult to dynamically expand or adjust. If new functions need to be added (such as intelligent analysis of people and objects in the program, or intelligent advertising recommendations), the existing solutions, which are fixed to the encoding process, are clearly unsuitable and inflexible. Therefore, we propose a novel IPTV architecture that transmits video streams and feature streams synchronously: the video stream is sent to the end-side nodes for decoding; the feature stream is sent to edge nodes, which use it for intelligent processing and recognition to provide ancillary services. The end-side nodes can then integrate and display these ancillary services with the video content, thereby enabling flexible expansion of ancillary services and meeting users' real-time interactive needs (this technology has been used in another patent application).

[0003] To adapt to the applications of auxiliary services in this architecture, the large number of video programs stored in the cloud need to undergo feature extraction to support the precoding of the feature stream. Although some video programs have undergone feature extraction due to their provision of various intelligent applications (such as intelligent recommendation and highlight clip editing), they are difficult to apply directly. This is because the designed IPTV architecture (edge ​​node-end node-cloud node architecture) requires a feature stream that corresponds one-to-one with the video frames of the video program. The features extracted by traditional feature extraction schemes do not meet the requirements. Therefore, a targeted feature extraction scheme needs to be designed to complement the IPTV architecture and provide a foundation for the flexible deployment of more intelligent auxiliary services.

[0004] However, directly using existing feature extraction schemes to extract full feature vectors frame-by-frame from IPTV video programs is too labor-intensive: the number of video programs stored on cloud nodes is enormous, and each video program contains a large number of video frames (for example, a 100-minute movie at 30 frames per second would have 6000 * 30 = 180,000 frames). Extracting full features frame-by-frame would consume excessive resources and be inefficient. Therefore, a more efficient feature extraction scheme suitable for this IPTV architecture needs to be designed. Summary of the Invention

[0005] The purpose of this application is to provide a video program feature extraction method based on artificial intelligence, so as to improve the extraction efficiency of IPTV video program feature streams and adapt to the new IPTV architecture.

[0006] To achieve the above objectives, the embodiments of this application are implemented in the following manner:

[0007] In a first aspect, embodiments of this application provide a video program feature extraction method based on artificial intelligence, applied to a cloud node, comprising: reading an IPTV video file stored on the cloud node and decoding it into an original frame sequence; performing shot decomposition on the original frame sequence to determine multiple shot segments; performing adaptive keyframe extraction on each shot segment, and determining sub-segments based on the keyframes, and then differentially extracting image features of each frame image in the sub-segments to obtain feature subsets corresponding to the sub-segments; integrating feature subsets belonging to the same shot segment to obtain a scene feature set; and integrating the scene feature sets corresponding to each shot segment to obtain a video feature set corresponding to the IPTV video file.

[0008] Beneficial Effects: This solution reads IPTV video files stored on cloud nodes, decodes them into raw frame sequences, deconstructs them into multiple shot segments, then performs adaptive keyframe extraction on each shot segment, and uses these keyframes as the core to determine sub-segments. Subsequently, it differentially extracts image features from each frame in the sub-segments (fully extracting feature vectors for keyframes, and differentially extracting feature vectors for non-keyframes in the sub-segments), obtaining feature subsets corresponding to the sub-segments. These feature subsets belonging to the same shot segment are then integrated to obtain a scene feature set. Finally, the scene feature sets corresponding to each shot segment are integrated to obtain the video feature set corresponding to the IPTV video file. This solution, through a layered processing logic of "shot deconstruction → keyframe extraction → sub-segment feature extraction," significantly reduces the number of frames requiring full feature extraction, focusing feature extraction on keyframes and their associated sub-segments, thus significantly reducing computational load. This feature extraction scheme ensures that features are strictly synchronized with video frames, guaranteeing that the features of each video frame can be derived by associating them with the key frames of its respective sub-segments. This satisfies the requirements of the end-edge-cloud collaborative architecture (i.e., the IPTV architecture of end-side nodes-edge nodes-cloud nodes) for frame-level synchronized feature streams.

[0009] In conjunction with the first aspect, in the first possible implementation of the first aspect, the original frame sequence is decomposed into multiple shot segments, including: downsampling the original frame sequence to obtain a preprocessed frame sequence, and performing color space conversion on the preprocessed frame sequence to obtain the corresponding Y-channel frame sequence and UV-channel frame sequence; performing luminance histogram statistics on the frame images in the Y-channel frame sequence, and performing chrominance histogram statistics on the frame images in the UV-channel frame sequence; based on the luminance histogram statistics results between the i-th frame image and the (i-1)-th frame image, calculating the luminance distribution difference corresponding to the i-th frame image, and based on the chrominance distribution differences between the i-th frame image and the (i-1)-th frame image... Histogram statistics are used to calculate the chromaticity distribution difference corresponding to the i-th frame image, where 1 < i ≤ n, and n is the total number of images in the original frame sequence. Optical flow analysis is performed on the i-th frame image and the (i-1)-th frame image in the Y-channel frame sequence to obtain the pixel motion index corresponding to the i-th frame image. The pixel motion index includes the average motion amplitude, motion direction consistency, and motion vector variance. Based on the luminance distribution difference, chromaticity distribution difference, and pixel motion index corresponding to the i-th frame image, it is determined whether the i-th frame image is a shot segmentation node, and all shot segmentation nodes are determined accordingly. Based on the shot segmentation nodes, the original frame sequence is decomposed into shots to determine multiple shot segments.

[0010] Beneficial Effects: During the lens segmentation process, three main categories of indicators are used to comprehensively determine the lens segmentation nodes: luminance histogram (calculating the difference in luminance distribution between adjacent frames), chrominance histogram (calculating the difference in chrominance distribution between adjacent frames), and optical flow analysis (motion amplitude, directional consistency, and vector variance). Multimodal fusion can more accurately identify lens transitions: luminance and chrominance are mainly used to capture abrupt changes in the overall color distribution of the image (such as scene transitions); while optical flow analysis is mainly used to detect drastic changes in local motion of the image (such as rapid transitions or object movement). Lens segmentation based on this can improve the accuracy of lens segmentation, reduce the false judgment rate, and adapt to both abrupt (such as hard cuts) and gradual (such as fade-in / fade-out) lens transition types. Considering the accuracy required for lens disassembly, downsampling the original frame sequence (spatial downsampling, such as reducing 1080P to 540P or even 360P) to generate a preprocessed frame sequence can effectively reduce the computational load of subsequent histogram statistics and optical flow analysis while meeting the lens disassembly analysis conditions (the loss of lens segmentation accuracy is small, while the consumption of computing resources is reduced by more than 70%, which is extremely cost-effective).

[0011] In conjunction with the first possible implementation of the first aspect, in the second possible implementation of the first aspect, based on the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to the i-th frame image, it is determined whether the i-th frame image is a shot segmentation node, and all shot segmentation nodes are determined accordingly, including: if the brightness distribution difference corresponding to the i-th frame image is higher than a set brightness distribution difference, the chromaticity distribution difference corresponding to the i-th frame image is higher than a set chromaticity distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than a first set value, the i-th frame image is marked as a candidate abrupt shot segmentation node; if the motion vector variance corresponding to x consecutive frames is greater than a second set value, and the cumulative difference calculated from the brightness distribution difference and chromaticity distribution difference corresponding to these x consecutive frames is higher than a set cumulative difference, these x consecutive frames are determined as a candidate gradual shot segmentation node interval, and candidate gradual shot segmentation nodes are determined from them; candidate abrupt shot segmentation nodes and candidate gradual shot segmentation nodes are integrated, and candidate shot segmentation nodes with an interval of less than a set number of frames between them and adjacent candidate shot segmentation nodes are removed to obtain all shot segmentation nodes.

[0012] Beneficial Effects: Based on different shot transition types, several shot segmentation node evaluation rules are designed: If the brightness distribution difference corresponding to the i-th frame image is higher than a set brightness distribution difference, the chromaticity distribution difference corresponding to the i-th frame image is higher than a set chromaticity distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than a first set value, the i-th frame image is marked as a candidate abrupt change shot segmentation node (suitable for rapid and accurate identification of abrupt change shots; by combining the brightness / chromaticity distribution difference threshold with the motion direction consistency index, it captures instantaneous changes in the color and structure of the image, such as hard cuts and black screen transitions, avoiding the omission of gradual scene transitions by a single detection logic; at the same time, low motion consistency indicates chaotic motion of the main subject in the image, which can filter out false abrupt change signals, such as flash interference); If the motion vector variance corresponding to x consecutive frames (e.g., 5 frames, 10 frames, etc.) is greater than a second set value, and the brightness distribution difference and chromaticity distribution difference corresponding to these x consecutive frames are ... identification of abrupt change If the calculated cumulative difference is higher than the set cumulative difference, the continuous x frames are determined as the candidate gradient shot segmentation node range, and candidate gradient shot segmentation nodes are determined from them (suitable for accurate identification of gradient shots, through the motion vector variance and cumulative color difference of continuous frames, such as fade-in / fade-out, dissolve effects, thus identifying the slow transition shot boundary, while high variance indicates that there is complex motion in the picture, such as rapid translation or rotation, which, together with the cumulative color difference, can enhance the reliability of gradient segmentation); then, candidate abrupt shot segmentation nodes and candidate gradient shot segmentation nodes are integrated, and candidate shot segmentation nodes with insufficient interval between adjacent candidate shot segmentation nodes (the set frame number here is converted from time to frame number, for example, the shortest shot of a general IPTV program will not be less than 2 seconds in length, if the frame rate is 30, this x is set to 60 frames, and if the frame rate is 60, this x needs to be set to 120) are eliminated, and all shot segmentation nodes are obtained. This allows for efficient and accurate segmentation of IPTV video programs at the shot level. By eliminating redundant shot segmentation nodes that are not spaced enough from adjacent nodes, it can effectively eliminate situations where there is brief jitter within the same shot rather than a real switch, thus avoiding over-segmentation.

[0013] In conjunction with the first possible implementation of the first aspect, in the third possible implementation of the first aspect, adaptive keyframe extraction is performed on each shot segment, including: for each shot segment: generating a sliding window of a set size, wherein the set size is less than a set number of frames; for the first sliding window of the current shot segment, taking the starting frame of the shot segment as the starting point: determining several keyframes by adopting a set inter-frame interval; for the non-first and last sliding windows of the current shot segment, taking the previous keyframe as the starting point: determining several keyframes within the sliding window based on the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to each frame image within the current sliding window; for the last sliding window of the current shot segment, taking the ending frame of the shot segment as the ending point: determining several keyframes by shifting the previous keyframe backward by a set inter-frame interval.

[0014] Beneficial Effects: Due to the diverse range of shot segments in IPTV video programs, fixed keyframe extraction strategies struggle to adapt to the varied dynamic shot scenes, making it difficult to select suitable keyframes for analysis. Therefore, this solution designs a sliding window analysis scheme. For each shot segment, sliding window analysis is employed (the sliding window size is such as 30 frames, 60 frames, etc.). The first and last windows use a scheme that determines keyframes by setting inter-frame intervals, capable of handling shot segments dominated by abrupt and gradual shot transitions simultaneously, ensuring coverage of the beginning and end phases of the shot segment and avoiding omissions of the beginning and end content. For the middle window of the shot segment (i.e., the non-beginning and non-end sliding window), starting from the previous keyframe, several keyframes within the sliding window are determined based on the differences in brightness distribution, chromaticity distribution, and pixel motion indicators corresponding to each frame image within the current sliding window. In non-first-and-last sliding window analysis, fixed-step sliding window analysis may lead to a large amount of redundant calculations, which is not conducive to accurately determining the appropriate keyframes. (In this scheme, the selection of keyframes is crucial because each subsequent keyframe will serve as the core of a sub-segment for full feature extraction. Other non-keyframes in the sub-segment centered on this keyframe will be extracted using differential feature vectors based on this keyframe. Therefore, the determination of keyframes will have a significant impact on the subsequent feature extraction process, and the extracted features will serve as the basis for auxiliary services in feature analysis of edge nodes.) Therefore, in non-first-and-last sliding window analysis, this scheme designs a dynamic sliding step and a comprehensive keyframe selection strategy. Utilizing the brightness distribution differences, chromaticity distribution differences, and pixel motion indicators calculated in the shot decomposition, appropriate evaluation schemes are designed according to different shot segment types (e.g., some are static shot segments, some are simple dynamic shot segments, and some are complex dynamic shots, etc.) as the basis for keyframe selection, thereby improving the accuracy of keyframe selection.

[0015] In conjunction with the third possible implementation of the first aspect, in the fourth possible implementation of the first aspect, several key frames within the sliding window are determined based on the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to each frame of the image in the current sliding window. This includes: calculating the motion state score corresponding to each frame of the image based on the pixel motion index corresponding to each frame of the image in the current sliding window; and calculating the color difference score corresponding to each frame of the image based on the brightness distribution difference and chromaticity distribution difference corresponding to each frame of the image. If there are images in the current sliding window whose motion state scores exceed the motion state threshold, a key frames are determined based on the motion state score and color difference score corresponding to each frame of the image. If there are no images in the current sliding window whose motion state scores exceed the motion state threshold, but there are images whose color difference scores exceed the color difference threshold, b key frames are determined based on the motion state score and color difference score corresponding to each frame of the image. If there are no images in the current sliding window whose motion state scores exceed the motion state threshold, and no images whose color difference scores exceed the color difference threshold, c key frames are determined based on the motion state score and color difference score corresponding to each frame of the image, where a>b>c>1.

[0016] Beneficial Effects: This solution designs motion state scoring and color difference scoring mechanisms for different scenarios (shot segments). Motion state scoring reflects the dynamic complexity of the image by quantifying average motion amplitude, directional consistency, and motion vector variance (e.g., motion state scores are significantly higher in fast-moving scenes). If an image with a motion state score exceeding the motion state threshold exists within the current sliding window, 'a' keyframes are determined based on the motion state score and color difference score corresponding to each frame (the number depends on the sliding window size; generally, for a 30-frame window, 'a' corresponds to 5 keyframes). Furthermore, combining brightness and chromaticity differences, a color difference score is calculated, which is beneficial for capturing sudden color changes in the image (such as changes in light and shadow). If no images with motion state scores exceeding the motion state threshold exist within the previous sliding window, but images with color difference scores exceeding the color difference threshold do exist, b keyframes are determined based on the motion state score and color difference score corresponding to each frame (for a window size of 30 frames, the corresponding number of keyframes b is 3). If neither image with a motion state score exceeding the motion state threshold nor an image with a color difference score exceeding the color difference threshold exists within the current sliding window, it indicates a relatively conventional, static shot. In this case, c keyframes are determined based on the motion state score and color difference score corresponding to each frame (for a window size of 30 frames, the corresponding number of keyframes c is 2), where a>b>c>1. Therefore, this scheme can densely sample within sliding windows with rapid motion or color changes (such as explosions or transitions) and sparsely sample in static areas (such as fixed shots), achieving on-demand resource allocation and more accurate keyframes, which is more conducive to high-quality feature extraction. Furthermore, by presetting the values ​​of parameters a, b, and c, the maximum number of keyframes in different scenes is constrained, avoiding the situation of unrestricted increase in keyframe sampling density.

[0017] In conjunction with the fourth possible implementation of the first aspect, in the fifth possible implementation of the first aspect, the motion state score corresponding to each frame of the image is calculated based on the pixel motion index corresponding to each frame of the image within the current sliding window, including: calculating the motion state score corresponding to the j-th frame of the image within the current sliding window using the following formula:

[0018] ,

[0019] in, Score the motion state corresponding to the j-th frame image within the current sliding window. , and For weight parameters, This represents the average motion amplitude corresponding to the j-th frame within the current sliding window. This is a reference value for the average amplitude of motion. To ensure the consistency of the motion direction corresponding to the j-th frame image within the current sliding window, Let $\mathbf{j}$ be the variance of the motion vector corresponding to the $j$-th frame image within the current sliding window. The motion vector variance reference value is used; based on the differences in brightness and chromaticity distributions corresponding to each frame, the color difference score corresponding to each frame is calculated, including: the color difference score corresponding to the j-th frame within the current sliding window is calculated using the following formula:

[0020] ,

[0021] in, Score the color difference corresponding to the j-th frame image within the current sliding window. The difference in brightness distribution corresponding to the j-th frame image within the current sliding window. The difference in chromaticity distribution corresponding to the j-th frame image within the current sliding window. This is the synergistic enhancement coefficient.

[0022] Beneficial effects: In the motion state scoring calculation scheme, logarithmic compression of the dimensional differences in motion amplitude (e.g., the amplitude value is extremely large in high-speed motion scenes) is used to prevent this indicator from easily dominating the motion state score; high motion consistency is transformed into low contribution value, highlighting indicators with chaotic direction (e.g., rapid camera shake or object collisions), while the motion vector variance is approximately standardized (not standardized, but converted using a motion vector variance reference value), and corresponding weight coefficients are assigned to each item (these can be empirically set parameter values ​​or learnable parameter values ​​to cope with different scenarios; however, considering that this would be more complex and increase the computational load, this scheme uses empirical setting as an example, while also considering different IPTV video program types and assigning different weights), flexibly adapting to different video content and improving the generalization ability when processing different IPTV video programs. The reference value is set based on empirical analysis, but global statistics can also be calculated in real time. The color difference scoring scheme addresses the fact that the differences in brightness and chromaticity distribution are not simply linearly related across different shot scenes, but rather exhibit a degree of synergy. When both brightness and chromaticity differences are high, the overall score should significantly improve (adapting to large changes in light and color within a shot segment). Conversely, in cases of a single factor (one high and one low in brightness and chromaticity distribution), the higher factor should dominate. For stable scenes, the score should be low, making them unsuitable as keyframes. Therefore, this color difference scoring scheme is designed to more accurately score the color differences of each frame, while the synergy enhancement coefficient serves as a means to balance the sensitivity to different types of shot segments, improving adaptability.

[0023] In conjunction with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, based on the motion state score and color difference score corresponding to each frame image, a keyframes are determined, including: calculating the comprehensive score corresponding to the j-th frame image within the current sliding window using the following formula:

[0024] ,

[0025] in, This represents the overall score corresponding to the j-th frame image within the current sliding window. The threshold for motion state is used; based on the comprehensive score corresponding to each frame image within the current sliding window, the a key frames with the highest comprehensive scores are determined.

[0026] Beneficial effect: For high-motion-type shot segments, motion state scoring This manifests as exceeding the motion state threshold. , As the score gradually approaches 1, the motion state score accounts for a higher proportion in the overall score, which can strengthen the selection of motion-driven keyframes.

[0027] In conjunction with the fifth possible implementation of the first aspect, in the seventh possible implementation of the first aspect, based on the motion state score and color difference score corresponding to each frame image, b keyframes are determined, including: calculating the comprehensive score corresponding to the j-th frame image within the current sliding window using the following formula:

[0028] ,

[0029] in, This is the overall score corresponding to the j-th frame image within the current sliding window. The color difference threshold is used; based on the comprehensive score corresponding to each frame image within the current sliding window, the b keyframes with the highest comprehensive scores are determined.

[0030] Beneficial effect: For low-motion-type shot segments, motion state scoring Not exceeding the motion state threshold At this point, a comprehensive calculation needs to be performed using both color difference score and motion state score, and... As a coefficient ( (Not zero), which allows the color difference weight to be adaptively enhanced according to the color difference score. In slow-moving shot segments, key frames can still be triggered by color differences, thereby determining more suitable key frames.

[0031] In conjunction with the fifth possible implementation of the first aspect, in the eighth possible implementation of the first aspect, based on the motion state score and color difference score corresponding to each frame image, c keyframes are determined, including: calculating the comprehensive score corresponding to the j-th frame image within the current sliding window using the following formula:

[0032] ,

[0033] in, The overall score is the score corresponding to the j-th frame image in the current sliding window; based on the overall score corresponding to each frame image in the current sliding window, the c keyframes with the highest overall scores are determined.

[0034] Beneficial effects: When both motion state score and color difference score are low, the two can be directly summed with equal weight (or other weights can be used). Generally, they will not be selected as keyframes. However, if all the frames in a sliding window are like this, the one with the highest overall score will be selected as the keyframe. (In this case, because the image changes little, the number of keyframes determined is small. Compared to other scenes with greater changes, in this case, one keyframe can dominate more non-keyframes and serve as the "backbone" for more subsequent non-keyframes.)

[0035] In conjunction with the fifth possible implementation of the first aspect, in the ninth possible implementation of the first aspect, sub-segments are determined with keyframes as the core, and then the image features of each frame in the sub-segment are extracted differentially to obtain the feature subset corresponding to the sub-segment. This includes: dividing the current shot segment into several sub-segments based on keyframes, with the first frame of each sub-segment being the keyframe, wherein each sub-segment does not contain keyframes other than the first frame; for each sub-segment: using ResNet50 to extract full feature vectors from the keyframes of the sub-segment, and, based on the keyframes of this sub-segment, using MobileNetV3 to extract differential feature vectors from each non-keyframe of the sub-segment, and then integrating the feature vectors based on the frame order of the sub-segment to obtain the feature subset corresponding to this sub-segment.

[0036] Beneficial effects: For each sub-segment, this solution extracts global features through ResNet50 full-frame processing and uses MobileNetV3 to extract differential features from non-key frames, which can effectively reduce the amount of processing. Furthermore, MobileNetV3 can be designed to be lightweight, further reducing computational overhead and making it suitable for large-scale video processing in the cloud.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart of an AI-based video program feature extraction method provided in this application embodiment.

[0040] Figure 2 A schematic diagram for determining the shot segmentation nodes.

[0041] Figure 3 A schematic diagram for determining keyframes.

[0042] Figure 4 This is a schematic diagram of the differential feature extraction process. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0044] Please see Figure 1 , Figure 1 The flowchart of the video program feature extraction method based on artificial intelligence provided in the embodiments of this application is applied to a cloud node and includes steps S10, S20, S30, S40 and S50.

[0045] In order to achieve feature extraction of IPTV video files stored on cloud nodes and ensure that each frame of the image has corresponding features, so as to adapt to the end-edge-cloud (end-side node-edge node-cloud node) IPTV architecture, this embodiment takes the feature extraction process of an IPTV video file as an example.

[0046] First, the cloud node can run step S10.

[0047] Step S10: Read the IPTV video file stored on the cloud node and decode it into the original frame sequence.

[0048] In this embodiment, the cloud node can read the stored IPTV video file to be processed and decode it into a raw frame sequence, the number of frames in the raw frame sequence being n. For example, the content of one episode of a TV series is 40 minutes long and has a frame rate of 30. The raw frame sequence contains 40*60*30=72000 frames, with a resolution of 1080P (i.e., 1920×1080 pixels). This is just an example; the actual resolution may be higher or lower. Video program content of other durations and frame rates is not limited.

[0049] After obtaining the original frame sequence, the cloud node can run step S20.

[0050] Step S20: Deconstruct the original frame sequence into multiple shot segments.

[0051] In this embodiment, the cloud node can downsample the original frame sequence (before this, the bottom area can be occluded or cropped, for example, the bottom 1 / 5 area, to avoid subtitles affecting the extraction of various indicators) to reduce the number of pixels in a single frame image, thereby significantly improving the processing speed while sacrificing a small amount of shot splitting accuracy) to obtain a preprocessed frame sequence. The number of frames in the preprocessed frame sequence is n, which is the same as the number of frames in the original frame sequence.

[0052] Then, the preprocessed frame sequence is converted to color space from RGB color space to YUV color space to obtain the corresponding Y channel frame sequence and UV channel frame sequence.

[0053] Based on this, luminance histogram statistics can be performed on the frame images in the Y-channel frame sequence, chromaticity histogram statistics can be performed on the frame images in the UV-channel frame sequence, and optical flow analysis can be performed based on the i-th frame image and the (i-1)-th frame image in the Y-channel frame sequence to obtain the pixel motion index corresponding to the i-th frame image. These three processes can be processed simultaneously to improve efficiency.

[0054] For example, the cloud node can perform luminance histogram statistics on the frame images in the Y-channel frame sequence. Then, based on the luminance histogram statistics of the i-th frame image and the (i-1)-th frame image, it calculates the luminance distribution difference corresponding to the i-th frame image (the absolute value of the luminance histogram statistics of the subsequent frame minus the luminance histogram statistics of the previous frame), where 1 < i ≤ n. For the first frame, the luminance histogram statistics can be used as the luminance distribution difference. In this embodiment, the first frame of the original frame sequence is to be used as the keyframe. Similarly, chrominance histogram statistics can be performed on the frame images in the UV-channel frame sequence. Based on the chrominance histogram statistics of the i-th frame image and the (i-1)-th frame image, it calculates the chrominance distribution difference corresponding to the i-th frame image (the absolute value of the chrominance histogram statistics of the subsequent frame minus the chrominance histogram statistics of the previous frame), where 1 < i ≤ n. Likewise, for the first frame, the chrominance histogram statistics can be used as the chrominance distribution difference. It should be noted that the brightness distribution difference corresponding to the i-th frame image here is calculated from the frame images in the Y channel frame sequence, and also corresponds to the i-th frame image in the original frame sequence; and the chromaticity distribution difference corresponding to the i-th frame image is calculated from the frame images in the UV channel frame sequence, and also corresponds to the i-th frame image in the original frame sequence.

[0055] Optical flow analysis takes the Y-channel frame sequence as input and outputs corresponding parameters. These parameters refer to the pixel motion metrics for the i-th frame (1 < i ≤ n), including average motion amplitude, motion direction consistency, and motion vector variance. For the first frame, the initial value is set to: average motion amplitude. Consistency of movement direction and the variance of motion vectors It is also important to note that the pixel motion index corresponding to the i-th frame image here is calculated from the frame images in the Y-channel frame sequence, and also corresponds to the i-th frame image in the original frame sequence.

[0056] The average motion amplitude corresponding to the i-th frame image:

[0057] , (1)

[0058] in, The average motion amplitude corresponding to the i-th frame image. The total number of feature points in the i-th frame of the Y-channel frame sequence. For the first frame of the image The optical flow method can use dense optical flow, where each pixel of the frame is treated as a feature point; or sparse optical flow, where a fixed subset of pixels is extracted from the frame as feature points; or the sum of pixels in a region is calculated as a single feature point. Each region, thus obtaining (Number of feature points, not limited here) For the first frame of the image The motion vectors of the feature points are calculated using the optical flow method. for The length of the module.

[0059] Consistency of motion direction corresponding to the i-th frame image:

[0060] , (2)

[0061] in, For the consistency of motion direction corresponding to the i-th frame image, For the first The direction angle of the motion vector of each feature point The main direction angle is obtained by taking the average direction angle of the motion vectors of all feature points in the image frame.

[0062] The variance of the motion vector corresponding to the i-th frame image:

[0063] , (3)

[0064] in, Let be the variance of the motion vector corresponding to the i-th frame image.

[0065] After obtaining the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to each frame of the image, the cloud node can determine whether the i-th frame image is a shot segmentation node based on the brightness distribution difference, chromaticity distribution difference, and pixel motion index corresponding to the i-th frame image, and thus determine all shot segmentation nodes.

[0066] During the shot breakdown process, three main categories of indicators are used to comprehensively determine shot segmentation nodes: luminance histogram (calculating the difference in luminance distribution between adjacent frames), chrominance histogram (calculating the difference in chrominance distribution between adjacent frames), and optical flow analysis (motion amplitude, directional consistency, and vector variance). Multimodal fusion can more accurately identify shot transitions: luminance and chrominance are mainly used to capture abrupt changes in the overall color distribution of the image (such as scene transitions); while optical flow analysis is mainly used to detect drastic changes in local motion of the image (such as rapid transitions or object movement). Based on this, shot breakdown can improve the accuracy of shot breakdown, reduce the false judgment rate, and adapt to both abrupt (such as hard cuts) and gradual (such as fade-in / fade-out) shot transition types. Considering the accuracy required for shot segmentation, downsampling the original frame sequence (spatial downsampling, such as reducing 1080P to 540P or even 360P; different downsampling indices can be determined for different types of IPTV video programs, for example, action-type programs such as sports events are reduced to 540P, and urban soap operas are reduced to 360P) to generate a preprocessed frame sequence can effectively reduce the computational load of subsequent histogram statistics and optical flow analysis while meeting the conditions for shot segmentation analysis (the loss of shot segmentation accuracy is small, while the consumption of computing resources is reduced by more than 70%, which is extremely cost-effective).

[0067] In this embodiment, based on different shot transition types (hard cut and fade), the following shot segmentation node evaluation rules are mainly designed:

[0068] If the brightness distribution difference corresponding to the i-th frame image is higher than the set brightness distribution difference, the chromaticity distribution difference corresponding to the i-th frame image is higher than the set chromaticity distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than the first set value, the cloud node can mark the i-th frame image as a candidate mutation lens segmentation node.

[0069] This approach is suitable for the rapid and accurate identification of sudden changes in scene (i.e., hard cuts). By combining the brightness / chromaticity distribution difference threshold with the motion direction consistency index, it captures instantaneous changes in the color and structure of the scene, such as hard cuts and black screen transitions. This avoids the omission of gradual scene changes by a single detection logic. At the same time, low motion consistency indicates chaotic movement of the main subject in the scene, which can filter out false sudden change signals, such as flash interference.

[0070] If the variance of the motion vector corresponding to x consecutive frames (e.g., 5 frames, 10 frames, etc.) is greater than the second set value, and the cumulative difference calculated from the difference in brightness distribution and the difference in chromaticity distribution corresponding to these x consecutive frames is higher than the set cumulative difference, these x consecutive frames are determined as candidate gradient lens segmentation node intervals, and candidate gradient lens segmentation nodes are determined from them.

[0071] This approach is suitable for the accurate identification of gradient shots. By using the motion vector variance of consecutive frames and the cumulative color difference, such as fade-in / fade-out and dissolve effects, the boundaries of slowly transitioning shots can be identified. High variance indicates that there is complex motion in the image, such as rapid translation or rotation. In conjunction with the cumulative color difference, it can enhance the reliability of gradient segmentation.

[0072] The cloud nodes can then integrate candidate abrupt change shot segmentation nodes and candidate gradual change shot segmentation nodes, eliminating candidate shot segmentation nodes whose interval with adjacent candidate shot segmentation nodes is less than a set number of frames, thus obtaining all shot segmentation nodes, such as... Figure 2 As shown, this schematically illustrates a small number of identified shot segmentation nodes.

[0073] The frame rate setting here is calculated from time to frame rate. For example, the shortest shot in a typical IPTV program is no less than 2 seconds long. If the frame rate is 30, then x is set to 60 frames, and if the frame rate is 60, then x needs to be set to 120. This allows for efficient and accurate segmentation of IPTV video programs at the shot level, while eliminating redundant shot segmentation nodes that are not spaced enough from adjacent nodes. This effectively eliminates situations where there is brief jitter within the same shot rather than a genuine transition, avoiding over-segmentation.

[0074] Accordingly, cloud nodes can decompose the original frame sequence into multiple shot segments based on the shot segmentation nodes. Note that this is shot decomposition of the original frame sequence, not a preprocessed frame sequence, nor a Y-channel frame sequence or a UV-channel frame sequence.

[0075] Once the shot segment is determined, the cloud node can proceed to step S30.

[0076] Step S30: Perform adaptive keyframe extraction for each shot segment, determine sub-segments based on the keyframes, and then extract the image features of each frame in the sub-segments in a differentiated manner to obtain the feature subset corresponding to the sub-segments.

[0077] Because IPTV video programs contain a wide variety of shot segments, fixed keyframe extraction strategies are ill-suited to the diverse dynamic shot scenes, making it difficult to select suitable keyframes for analysis. Therefore, this solution designs a sliding window analysis scheme. In this embodiment, for each shot segment:

[0078] Cloud nodes can generate sliding windows of a set size, where the set size is smaller than a set number of frames. For example, the sliding window size is designed to be 30 frames (or 60 frames, or other sizes).

[0079] For the first sliding window of the current shot segment (e.g., a 300-frame shot segment), starting from the beginning frame of the shot segment, several keyframes are determined using a set inter-frame interval (e.g., 5 frames). These keyframes are frame 1, frame 7, frame 13, frame 19, and frame 25. For the last sliding window of the current shot segment (where the time from the previous keyframe to the end frame of the shot segment is exactly 30 frames or less), starting from the end frame of the shot segment, several keyframes are determined by shifting backward from the previous keyframe by a set inter-frame interval (again, 5 frames per second).

[0080] The beginning and end windows use a scheme that sets the inter-frame interval to determine keyframes, which can simultaneously handle shot segments dominated by abrupt and gradual shot transitions, ensuring coverage of the beginning and end phases of shot segments and avoiding omissions of beginning and end content.

[0081] For a non-first and last sliding window of the current shot segment, the cloud node needs to take the previous keyframe as the starting point: based on the differences in brightness distribution, color distribution, and pixel motion indicators of each frame in the current sliding window, several keyframes in the sliding window are determined.

[0082] In non-first-and-last sliding window analysis, fixed-step sliding window analysis may lead to a large amount of redundant calculations, which is not conducive to accurately determining the appropriate keyframes (in this embodiment, the selection of keyframes is crucial because each subsequent keyframe will serve as the core of a sub-segment for full feature extraction, while other non-keyframes in the sub-segment centered on that keyframe will be used as the benchmark for differential feature vector extraction. Therefore, the determination of keyframes will have a significant impact on the subsequent feature extraction process, and the extracted features will serve as the basis for auxiliary services in feature analysis of edge nodes). Therefore, in non-first-and-last sliding window analysis, this scheme designs a dynamic sliding step and a comprehensive keyframe selection strategy. Utilizing the brightness distribution differences, chromaticity distribution differences, and pixel motion indicators calculated in the shot decomposition, appropriate evaluation schemes are designed according to different shot segment types (e.g., some are tending to be static shot segments, some are simple dynamic shot segments, and some are complex dynamic shots, etc.) as the basis for keyframe selection, thereby improving the accuracy of keyframe selection.

[0083] For example, a cloud node can calculate the motion state score for each frame of an image based on the pixel motion index corresponding to each frame of an image within the current sliding window.

[0084] Specifically, the cloud node uses the following formula to calculate the motion state score corresponding to the j-th frame image within the current sliding window:

[0085] , (4)

[0086] in, Score the motion state of the j-th frame image (with a unique original frame number) within the current sliding window. , and For weight parameters, This represents the average motion amplitude corresponding to the j-th frame within the current sliding window. This is a reference value for the average amplitude of motion. To ensure the consistency of the motion direction corresponding to the j-th frame image within the current sliding window, Let $\mathbf{j}$ be the variance of the motion vector corresponding to the $j$-th frame image within the current sliding window. This is the reference value for the variance of the motion vector.

[0087] In the motion state scoring calculation scheme, logarithmic compression is used to reduce the dimensional differences in motion amplitude (e.g., the amplitude value is extremely large in high-speed motion scenes), preventing this indicator from easily dominating the motion state score. High motion consistency is transformed into a low contribution value, highlighting indicators with chaotic direction (e.g., rapid camera shake or object collisions). The motion vector variance is approximately standardized (not truly standardized, but converted using a motion vector variance reference value), and corresponding weight coefficients are assigned to each item (these can be empirically set parameter values ​​or learnable parameter values ​​to cope with different scenarios; however, considering that this would be more complex and increase computational load, this scheme uses empirical setting as an example, while also considering different IPTV video program types and assigning different weights). This flexibly adapts to different video content and improves the generalization ability when processing different IPTV video programs. The reference value is set based on empirical analysis, but global statistics can also be calculated in real time.

[0088] Furthermore, cloud nodes can calculate the color difference score for each frame of an image based on the differences in brightness and chromaticity distribution.

[0089] Specifically, the color difference score corresponding to the j-th frame image within the current sliding window is calculated using the following formula:

[0090] , (5)

[0091] in, Score the color difference corresponding to the j-th frame image within the current sliding window. The difference in brightness distribution corresponding to the j-th frame image within the current sliding window. The difference in chromaticity distribution corresponding to the j-th frame image within the current sliding window. This is the synergistic enhancement coefficient.

[0092] In the color difference scoring scheme, since the differences in brightness and chromaticity distribution are not simply linear in different shot scenes, but rather exhibit a certain degree of synergy, when both brightness and chromaticity distribution differences are high, the overall score should be significantly improved (to accommodate large changes in light and color within a shot segment); while in the case of a single factor (one brightness and one chromaticity distribution difference being high and the other low), the higher factor should dominate; for stable scenes, the score should be low, making them unsuitable as keyframes. Therefore, this color difference scoring scheme was designed to more accurately score the color differences of each frame, while the synergy enhancement coefficient can serve as a means to balance the sensitivity to different types of shot segments, improving adaptability to different types of shot segments.

[0093] After calculating the motion state score and color difference score for each frame within the current sliding window, the cloud node can make judgments and perform corresponding processing:

[0094] If there are images in the current sliding window whose motion state scores exceed the motion state threshold (to improve reliability, the judgment condition can be set more strictly, for example, if there are at least 3 images in the current sliding window whose motion state scores exceed the motion state threshold), then a key frames are determined based on the motion state score and color difference score corresponding to each frame.

[0095] Specifically, the cloud node can use the following formula to calculate the comprehensive score corresponding to the j-th frame image within the current sliding window:

[0096] , (6)

[0097] in, This represents the overall score corresponding to the j-th frame image within the current sliding window. The motion state threshold is defined as follows (0.7 for sports events and 0.5 for film and television programs). For high-motion-type shot segments, the motion state score is... This manifests as exceeding the motion state threshold. , As the score gradually approaches 1, the motion state score accounts for a higher proportion in the overall score, which can strengthen the selection of motion-driven keyframes.

[0098] Based on this, the cloud node can determine the 'a' keyframes with the highest comprehensive scores based on the comprehensive score corresponding to each frame image in the current sliding window (for a sliding window of size 30 frames, the number of keyframes 'a' is 5 for example).

[0099] If there are no images in the current sliding window whose motion state score exceeds the motion state threshold (correspondingly, in the reliability judgment condition, it is designed that: there are less than 3 images in the current sliding window whose motion state score exceeds the motion state threshold), but there are images whose color difference score exceeds the color difference threshold (similarly designed that there are at least 3 images in the current sliding window whose color difference score exceeds the color difference threshold), then the cloud node can determine b keyframes based on the motion state score and color difference score corresponding to each frame of the image.

[0100] Specifically, the cloud node can use the following formula to calculate the comprehensive score corresponding to the j-th frame image within the current sliding window:

[0101] , (7)

[0102] in, This is the overall score corresponding to the j-th frame image within the current sliding window. Color difference threshold ( The value is not zero, and is generally between 0.3 and 0.4. For special scenes such as fireworks displays and light shows, the value needs to be increased, and is not expected to be lower than 0.5. For low-motion footage, the motion state score... Not exceeding the motion state threshold At this point, a comprehensive calculation needs to be performed using both color difference score and motion state score, and... As a coefficient, the color difference weight can be adaptively enhanced according to the color difference score. Even in slow-moving shot segments, key frames can still be triggered by color differences, thereby determining more suitable key frames.

[0103] Accordingly, the cloud node can determine the b keyframes with the highest comprehensive scores (b is 3 in this embodiment) based on the comprehensive score corresponding to each frame of the image in the current sliding window.

[0104] If there are no images in the current sliding window whose motion state score exceeds the motion state threshold (correspondingly, in the reliability judgment condition, it is designed that: there are less than 3 images in the current sliding window whose motion state score exceeds the motion state threshold), and there are no images whose color difference score exceeds the color difference threshold (there are less than 3 images in the current sliding window whose color difference score exceeds the color difference threshold), then the cloud node can determine c keyframes based on the motion state score and color difference score corresponding to each frame of the image, where a>b>c>1.

[0105] Specifically, the cloud node can use the following formula to calculate the comprehensive score corresponding to the j-th frame image within the current sliding window:

[0106] , (8)

[0107] in, This is the overall score corresponding to the j-th frame image within the current sliding window. For cases where both the motion score and color difference score are low, they can be directly summed using a weighted average (or other weighting methods can be used). Generally, this will not be selected as a keyframe. However, if all frames within a sliding window exhibit this situation, the one with the highest overall score will be selected as the keyframe. (In this case, because the image changes little, the number of keyframes is relatively small. Compared to other scenes with greater changes, in this case, one keyframe can dominate more non-keyframes, serving as the "backbone" for subsequent non-keyframes.)

[0108] Accordingly, the cloud node can determine the c keyframes with the highest comprehensive scores (in this embodiment, c is 2) based on the comprehensive score corresponding to each frame of the image in the current sliding window.

[0109] For different scenarios (shot segments), this embodiment designs a motion state scoring and color difference scoring mechanism. The motion state scoring reflects the dynamic complexity of the image by quantifying the average motion amplitude, directional consistency, and motion vector variance (e.g., the motion state score is significantly higher in fast-moving scenes). If there is an image in the current sliding window whose motion state score exceeds the motion state threshold, based on the motion state score and color difference score corresponding to each frame, 'a' keyframes are determined (the number depends on the size of the sliding window; generally, for a 30-frame window, 'a' corresponds to 5 keyframes). The color difference score is calculated by combining brightness and chromaticity differences, which is beneficial for capturing sudden color changes in the image (such as changes in light and shadow). If the current sliding window contains an image with a motion state score exceeding the motion state threshold, then 'a' keyframes are determined based on the motion state score and color difference score corresponding to each frame (the number of keyframes is determined based on the size of the sliding window; generally, for a 30-frame window, 'a' corresponds to 5 keyframes). If no image with a motion state score exceeding the motion state threshold exists within the sliding window, but an image with a color difference score exceeding the color difference threshold does exist, b keyframes are determined based on the motion state score and color difference score corresponding to each frame (for a window size of 30 frames, the corresponding number of keyframes b is 3). If neither an image with a motion state score exceeding the motion state threshold nor an image with a color difference score exceeding the color difference threshold exists within the current sliding window, it indicates a relatively conventional, static shot. In this case, c keyframes are determined based on the motion state score and color difference score corresponding to each frame (for a window size of 30 frames, the corresponding number of keyframes c is 2), where a>b>c>1. Therefore, this scheme can densely sample within sliding windows with rapid motion or color changes (such as explosions or transitions) and sparsely sample in static areas (such as fixed shots), achieving on-demand resource allocation and more accurate keyframes, which is more conducive to high-quality feature extraction. Furthermore, by presetting the values ​​of parameters a, b, and c, the maximum number of keyframes in different scenarios is constrained, avoiding the situation of unrestricted increase in keyframe sampling density.

[0110] After identifying the keyframes, the cloud nodes can deduplicate them and then use the keyframes as the core to determine sub-segments. Specifically, the current shot segment is divided into several sub-segments using the keyframes as nodes. The first frame of each sub-segment is the keyframe, and each sub-segment does not contain any keyframes other than the first frame. Figure 3 As shown.

[0111] After determining the sub-segments using keyframes as the core, cloud nodes can extract the image features of each frame in the sub-segments in a differentiated manner to obtain the feature subset corresponding to the sub-segments.

[0112] For example, for each sub-segment: the cloud node can use ResNet50 to extract full feature vectors from the key frames of the sub-segment (e.g., the extracted full feature vectors are 1024-dimensional, corresponding to a frame in the original frame sequence). Based on the key frames of this sub-segment, MobileNetV3 can be used to extract differential feature vectors from each non-key frame of the sub-segment (e.g., the extracted full feature vectors are 256-dimensional, also corresponding to each frame in the original frame sequence). Then, feature vectors are integrated based on the frame order of the sub-segment (integration can be performed according to the true order) to obtain the feature subset corresponding to this sub-segment.

[0113] For details on the differential feature extraction process, please refer to [link / reference]. Figure 4 The keyframe (first frame) of this sub-segment is used as the reference frame. Optical flow estimation is performed on each non-keyframe of this sub-segment to obtain the optical flow field (pixel-level motion information, i.e., the horizontal and vertical displacement of each pixel). This optical flow field is then used for motion compensation alignment (i.e., alignment of non-keyframes with keyframes). Further, a difference image between the non-keyframes and keyframes is obtained and input into MobileNetV3 (a lightweight version, such as MobileNetV3-Small, can be selected). The 256-dimensional difference feature vector of the non-keyframe is extracted and output. It should be noted that the vector dimensions of keyframes and non-keyframes are different, which does not affect subsequent applications. Furthermore, differential coding can be used for pre-coding, eliminating the need for actual encoding during transmission and improving the operational efficiency of the IPTV edge-cloud architecture.

[0114] This solution extracts global features through full-frame processing using ResNet50 and performs differential feature extraction on non-key frames using MobileNetV3, which effectively reduces the amount of processing. Furthermore, MobileNetV3 can be designed to be lightweight, further reducing computational overhead and making it suitable for large-scale video processing in the cloud.

[0115] After obtaining the feature subset corresponding to each sub-segment, the cloud node can run step S40.

[0116] Step S40: Integrate feature subsets belonging to the same shot segment to obtain scene feature set.

[0117] In this embodiment, cloud nodes can integrate feature subsets belonging to the same shot segment according to frame order to obtain a scene feature set (or shot feature set).

[0118] Afterwards, the cloud node can run step S50.

[0119] Step S50: Integrate the scene feature sets corresponding to each shot segment to obtain the video feature set corresponding to the IPTV video file.

[0120] In this embodiment, the cloud node can integrate the scene feature set corresponding to each shot segment according to the frame order to obtain the video feature set corresponding to the IPTV video file. The number of features in the video feature set is the same as the number of frames in the original frame sequence in the IPTV video file, and they can correspond one-to-one.

[0121] In summary, this application provides an AI-based video program feature extraction method. It reads IPTV video files stored on cloud nodes, decodes them into raw frame sequences, deconstructs them into multiple shot segments, performs adaptive keyframe extraction on each shot segment, and identifies sub-segments based on these keyframes. Then, it differentially extracts image features from each frame within the sub-segments (full feature vector extraction for keyframes, and differential feature vector extraction for non-keyframes in the sub-segments), obtaining feature subsets corresponding to the sub-segments. These feature subsets belonging to the same shot segment are integrated to obtain a scene feature set. Finally, the scene feature sets corresponding to each shot segment are integrated to obtain the video feature set corresponding to the IPTV video file. This approach, through a hierarchical processing logic of "shot deconstruction → keyframe extraction → sub-segment feature extraction," significantly reduces the number of frames requiring full feature extraction, focusing feature extraction on keyframes and their associated sub-segments, thus significantly reducing computational load. This feature extraction scheme ensures that features are strictly synchronized with video frames, guaranteeing that the features of each video frame can be derived by associating them with the key frames of its respective sub-segments. This satisfies the requirements of the end-edge-cloud collaborative architecture (i.e., the IPTV architecture of end-side nodes-edge nodes-cloud nodes) for frame-level synchronized feature streams.

[0122] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video program feature extraction method based on artificial intelligence, characterized in that, Applied to cloud nodes, including: Read IPTV video files stored on cloud nodes and decode them into raw frame sequences; The original frame sequence was broken down into shots to identify multiple shot segments; Adaptive keyframe extraction is performed on each shot segment, and sub-segments are determined with keyframes as the core. Then, the image features of each frame in the sub-segment are extracted differentially to obtain the feature subset corresponding to the sub-segment. Integrate feature subsets belonging to the same shot segment to obtain the scene feature set; By integrating the scene feature sets corresponding to each shot segment, the video feature set corresponding to the IPTV video file is obtained; The original frame sequence was broken down into shots, identifying multiple shot segments, including: The original frame sequence is downsampled to obtain a preprocessed frame sequence, and the preprocessed frame sequence is then color space converted to obtain the corresponding Y channel frame sequence and UV channel frame sequence. Luminance histogram statistics are performed on the frame images in the Y channel frame sequence, and chrominance histogram statistics are performed on the frame images in the UV channel frame sequence; Based on the luminance histogram statistics of the i-th frame image and the (i-1)-th frame image, calculate the luminance distribution difference corresponding to the i-th frame image. Based on the chrominance histogram statistics of the i-th frame image and the (i-1)-th frame image, calculate the chrominance distribution difference corresponding to the i-th frame image. 1 < i ≤ n, where n is the total number of images in the original frame sequence. Optical flow analysis is performed on the i-th frame image and the (i-1)-th frame image in the Y-channel frame sequence to obtain the pixel motion index corresponding to the i-th frame image. The pixel motion index includes the average motion amplitude, motion direction consistency and motion vector variance. Based on the differences in brightness distribution, chromaticity distribution, and pixel motion indicators corresponding to the i-th frame image, determine whether the i-th frame image is a shot segmentation node, and thus determine all shot segmentation nodes. Based on the shot segmentation nodes, the original frame sequence is decomposed into shots to determine multiple shot segments; Adaptive keyframe extraction is performed on each shot segment, including: For each shot / segment: Generate a sliding window of a set size, where the set size is smaller than a set number of frames; For the first sliding window of the current shot segment, starting from the beginning frame of the shot segment, a number of keyframes are determined using the set inter-frame interval; For the non-first and last sliding window of the current shot segment, starting from the previous keyframe: based on the differences in brightness distribution, color distribution, and pixel motion indicators of each frame in the current sliding window, several keyframes in the sliding window are determined. For the last sliding window of the current shot segment, with the end frame of the shot segment as the endpoint: a number of keyframes are determined by pushing forward the set inter-frame interval from the previous keyframe. Based on the differences in brightness distribution, chromaticity distribution, and pixel motion metrics corresponding to each frame within the current sliding window, several key frames within the sliding window are determined, including: Based on the pixel motion index corresponding to each frame of the image within the current sliding window, the motion state score corresponding to each frame of the image is calculated. Based on the brightness distribution difference and chromaticity distribution difference corresponding to each frame of the image, the color difference score corresponding to each frame of the image is calculated. If there is an image in the current sliding window whose motion state score exceeds the motion state threshold, determine a keyframes based on the motion state score and color difference score corresponding to each frame image; If there are no images with motion state scores exceeding the motion state threshold within the current sliding window, but there are images with color difference scores exceeding the color difference threshold, b keyframes are determined based on the motion state score and color difference score corresponding to each frame image. If there are no images in the current sliding window whose motion state score exceeds the motion state threshold and no images whose color difference score exceeds the color difference threshold, then based on the motion state score and color difference score corresponding to each frame image, determine c keyframes, where a>b>c>1. Based on the pixel motion metrics corresponding to each frame within the current sliding window, a motion state score is calculated for each frame, including: The motion state score corresponding to the j-th frame image within the current sliding window is calculated using the following formula: ， in, Score the motion state corresponding to the j-th frame image within the current sliding window. , and For weight parameters, This represents the average motion amplitude corresponding to the j-th frame within the current sliding window. This is a reference value for the average amplitude of motion. To ensure consistency of the motion direction corresponding to the j-th frame image within the current sliding window, Let $\mathbf{j}$ be the variance of the motion vector corresponding to the $j$-th frame image within the current sliding window. This is a reference value for the variance of the motion vector; Based on the differences in luminance and chromaticity distributions corresponding to each frame of the image, a color difference score is calculated for each frame, including: The color difference score corresponding to the j-th frame image within the current sliding window is calculated using the following formula: ， in, Score the color difference corresponding to the j-th frame image within the current sliding window. The difference in brightness distribution corresponding to the j-th frame image within the current sliding window. The difference in chromaticity distribution corresponding to the j-th frame image within the current sliding window. This is the synergistic enhancement coefficient.

2. The video program feature extraction method based on artificial intelligence according to claim 1, characterized in that, Based on the differences in brightness distribution, chromaticity distribution, and pixel motion indicators corresponding to the i-th frame image, it is determined whether the i-th frame image is a shot segmentation node. Based on this, all shot segmentation nodes are identified, including: If the brightness distribution difference corresponding to the i-th frame image is higher than the set brightness distribution difference, the chromaticity distribution difference corresponding to the i-th frame image is higher than the set chromaticity distribution difference, and the motion direction consistency in the pixel motion index corresponding to the i-th frame image is lower than the first set value, then the i-th frame image is marked as a candidate mutation lens segmentation node. If the variance of the motion vector corresponding to x consecutive frames is greater than the second set value, and the cumulative difference calculated from the difference in brightness distribution and the difference in chromaticity distribution corresponding to x consecutive frames is higher than the set cumulative difference, then the x consecutive frames are determined as candidate gradient lens segmentation node intervals, and candidate gradient lens segmentation nodes are determined from them. By integrating candidate mutation shot segmentation nodes and candidate gradient shot segmentation nodes, and removing candidate shot segmentation nodes whose interval with adjacent candidate shot segmentation nodes is less than a set number of frames, all shot segmentation nodes are obtained.

3. The video program feature extraction method based on artificial intelligence according to claim 1, characterized in that, Based on the motion state score and color difference score corresponding to each frame of the image, a key frames are determined, including: The overall score corresponding to the j-th frame image within the current sliding window is calculated using the following formula: ， in, This represents the overall score corresponding to the j-th frame image within the current sliding window. The threshold for motion state; Based on the comprehensive score corresponding to each frame image within the current sliding window, the a key frames with the highest comprehensive scores are determined.

4. The video program feature extraction method based on artificial intelligence according to claim 1, characterized in that, Based on the motion state score and color difference score corresponding to each frame of the image, b key frames are determined, including: The overall score corresponding to the j-th frame image within the current sliding window is calculated using the following formula: ， in, This is the overall score corresponding to the j-th frame image within the current sliding window. The color difference threshold; Based on the comprehensive score corresponding to each frame image within the current sliding window, the b keyframes with the highest comprehensive scores are determined.

5. The video program feature extraction method based on artificial intelligence according to claim 1, characterized in that, Based on the motion state score and color difference score corresponding to each frame of the image, c key frames are determined, including: The overall score corresponding to the j-th frame image within the current sliding window is calculated using the following formula: ， in, This is the overall score corresponding to the j-th frame image within the current sliding window; Based on the comprehensive score corresponding to each frame image within the current sliding window, the c keyframes with the highest comprehensive scores are determined.

6. The video program feature extraction method based on artificial intelligence according to claim 1, characterized in that, Using keyframes as the core, sub-segments are determined, and then image features of each frame in the sub-segment are extracted differentially to obtain the feature subset corresponding to the sub-segment, including: The current shot is divided into several sub-segments based on keyframes. The first frame of each sub-segment is a keyframe. Each sub-segment does not contain any keyframes other than the first frame. For each sub-segment: ResNet50 is used to extract full feature vectors from the key frames of the sub-segment. Based on the key frames of this sub-segment, MobileNetV3 is used to extract differential feature vectors from each non-key frame of the sub-segment. Then, feature vectors are integrated based on the frame order of the sub-segment to obtain the feature subset corresponding to this sub-segment.

Images