Video shot type identification method and system based on picture analysis principle

Abstract

The invention discloses a video shot type identification method and system based on a picture analysis principle, and the method comprises the steps: extracting a video frame from a preprocessed video file, and extracting a key frame from the video frame through a content-based frame difference analysis method; respectively extracting and describing visual features of the key frame by using a corresponding feature extraction method; the visual features comprise color features, texture features, shape features and motion features; establishing a feature extraction model for each shot type based on the visual features of the video file key frame according to the definition of the shot type; feature vectors of different lens types are analyzed and learned; and analyzing the feature vectors of the lens types by using the trained classifier, and identifying the lens types by calculating the matching degree between the feature vectors of the lens types and the feature models of the corresponding lens types. The method has the advantages that a user can be helped to quickly understand the content structure and characteristics of the video, and rich information is provided for further understanding the video content.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video shot type recognition method and system based on picture analysis principle. Background Art

[0002] For complex scenes and fast-changing shots, existing methods may not be accurate enough in feature extraction, resulting in classification errors and difficulty in accurately identifying the shot type. Existing technologies require special training and adjustments for different video types and lack universality. Summary of the invention

[0003] The object of the present invention is to provide a method and system for identifying video shot types based on picture analysis principles, thereby solving the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A video shot type recognition method based on picture analysis principle comprises the following steps:

[0006] S1. Video frame extraction and key frame selection: extract video frames from the preprocessed video files, and extract key frames from the video frames through a content-based frame difference analysis method;

[0007] S2, visual feature extraction: using the corresponding feature extraction method to extract and describe the visual features of the key frames respectively; the visual features include color features, texture features, shape features and motion features;

[0008] S3, shot type feature modeling: According to the definition of shot type, based on the visual features of the key frames of the video file, a feature extraction model is established for each shot type; and the feature vectors of different shot types are analyzed and learned;

[0009] S4. Lens type identification: Analyze the feature vector of the lens type using the trained classifier, and identify the lens type by calculating the matching degree between the feature vector of each lens type and the corresponding lens type feature model.

[0010] Preferably, step S1 specifically includes the following contents:

[0011] S11, extracting video frames from the video file at fixed time intervals, or determining the number of extracted video frames according to the frame rate of the video file; and storing the extracted video frames;

[0012] S12, the content-based frame difference analysis method determines the key frame in the video frame by comparing the content difference between adjacent video frames, and if the content difference value between adjacent video frames is greater than a preset threshold, it is determined that there is a key frame in the adjacent video frame;

[0013] S13, further filter the key frames based on the time interval and the motion amplitude.

[0014] Preferably, step S2 specifically includes the following contents:

[0015] S21, extracting and describing the color features of the key frame by extracting a color histogram and a color moment;

[0016] S22, extracting and describing texture features of the key frame using a gray level co-occurrence matrix or a local binary pattern method;

[0017] S23, using edge detection, Hough transform, and Fourier descriptor methods to extract and describe the shape features of the key frames;

[0018] S24. Extract and describe the motion features of the key frames by calculating optical flow and motion vectors.

[0019] Preferably, step S3 specifically includes the following contents:

[0020] S31. Identify different types of lenses and, for each lens type, determine the characteristics it may have;

[0021] S32, analyzing the visual features of key frames of a large number of video files of known shot types, summarizing the typical features of each shot type, and establishing a feature model for each shot type;

[0022] S33. Analyze and learn the feature vectors of different shot types, including motion speed, scene depth, and the relationship between the subject and the background.

[0023] Preferably, step S4 specifically includes the following contents:

[0024] S41, substituting the feature vector extracted in step S3 into the hyperplane equation to obtain the position of the hyperplane;

[0025] S42, preliminarily determining the lens type according to the calculated value, if the value is greater than or equal to 0, the feature vector is determined to be a positive class, otherwise it is determined to be a negative class;

[0026] S43, performing a similarity comparison between the extracted feature vector and the lens type feature model of the corresponding class to measure the similarity between the feature vector and each lens type feature model;

[0027] S44, selecting the lens type with the highest similarity as the recognition result.

[0028] Preferably, after step S4, the method further includes:

[0029] S5. Result analysis and optimization: Analyze the recognition results, evaluate the recognition accuracy, and adjust the parameters of the feature extraction method and / or classifier based on the feedback of the recognition results, or directly replace the feature extraction method and / or classifier to optimize the recognition performance.

[0030] Preferably, step S5 specifically includes the following contents:

[0031] S51, comparing the recognition result with the result of manual annotation, and using multiple evaluation indicators to evaluate the accuracy of recognition; analyzing the case of misplaced recognition and determining the cause of misplaced recognition;

[0032] S52. If the reason for misalignment recognition is inaccurate feature extraction, adjust the parameters of the feature extraction algorithm, or try to use other feature extraction algorithms; if the reason for misalignment recognition is poor classifier performance, adjust the parameters of the classifier, or try to use other classifiers or integrate multiple classifiers.

[0033] Preferably, before step S1, the step further includes:

[0034] S0, video input and preprocessing: denoising, frame rate adjustment, and resolution standardization operations are performed on the input video files.

[0035] Preferably, step S0 specifically includes the following contents:

[0036] S01, receiving video files of various formats as input data;

[0037] S02. Use median filtering and Gaussian filtering to effectively remove noise interference in video files;

[0038] S03, adjusting the frame rate of the video file to an appropriate value according to subsequent processing requirements and hardware performance;

[0039] S04. Standardize the resolution of the video files to unify the resolution of each video file.

[0040] The present invention also aims to provide a video shot type recognition system based on the picture analysis principle, the system can implement the above-mentioned method, the system comprises:

[0041] Video input and preprocessing module: denoise, adjust frame rate, and standardize resolution of input video files

[0042] Video frame extraction and key frame selection module: extracts video frames from the preprocessed video files, and extracts key frames from the video frames through content-based frame difference analysis method;

[0043] Visual feature extraction module: Use corresponding feature extraction methods to extract and describe the visual features of key frames respectively; visual features include color features, texture features, shape features and motion features;

[0044] Shot type feature modeling module: According to the definition of shot type, based on the visual features of the key frames of the video file, a feature extraction model is established for each shot type; and the feature vectors of different shot types are analyzed and learned;

[0045] Lens type identification module: Use the trained classifier to analyze the feature vector of the lens type, and identify the lens type by calculating the matching degree between the feature vector of each lens type and the corresponding lens type feature model.

[0046] Result analysis and optimization module: analyzes the recognition results, evaluates the recognition accuracy, and adjusts the parameters of the feature extraction method and / or classifier based on the feedback of the recognition results, or directly replaces the feature extraction method and / or classifier to optimize the recognition performance.

[0047] User interface: used by users to upload video files, start the recognition process, and display the recognition results.

[0048] The beneficial effects of the present invention are: 1. The present invention can accurately identify different types of shots through multiple feature extractions and advanced classifiers, greatly reducing the possibility of misidentification; the optimization algorithm meets the real-time processing requirements, reduces computational complexity, and improves processing speed. 2. The present invention analyzes and learns different types of features, and establishes a feature model for each type of shot, so that the present invention can adapt to different types of video content and shooting styles, providing good versatility and adaptability. 3. The user interface developed by the present invention allows users to upload videos, and the interactive functions provided enable users to better participate in and adjust the recognition process, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of an identification method in an embodiment of the present invention;

[0050] Figure 2 is a flow chart of video preprocessing in an embodiment of the present invention;

[0051] Figure 3 is a flowchart of key frame selection in an embodiment of the present invention;

[0052] Figure 4 is a flow chart of feature modeling in an embodiment of the present invention;

[0053] Figure 5 4 is a flowchart of lens type identification in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] like Figure 1 As shown, in this embodiment, a video shot type identification method based on the picture analysis principle is provided, including the following parts:

[0056] 1. Video input and preprocessing

[0057] Receive video files as input and preprocess them, including denoising, frame rate adjustment, resolution standardization, etc., to improve the efficiency and accuracy of subsequent processing. Figure 2 As shown, it specifically includes the following contents:

[0058] 1.1. Video file input: Receive video files of various formats as input data.

[0059] 1.2 Denoising: Use median filtering, Gaussian filtering and other techniques to effectively remove noise interference in the video.

[0060] 1.3 Frame rate adjustment: Adjust the frame rate to an appropriate value based on subsequent processing requirements and hardware performance.

[0061] 1.4 Resolution Standardization: Carry out resolution standardization to unify the video resolution to improve the efficiency and accuracy of subsequent processing. When adjusting the resolution, an interpolation algorithm can be used to maintain the quality of the image.

[0062] 2. Video frame extraction and key frame selection

[0063] Extract video frames from the preprocessed video file. Select key frames through algorithms (such as content-based frame difference analysis), which can represent the key change points of the video content. Figure 3 As shown, it specifically includes the following contents:

[0064] 2.1. Frame extraction: Frames are extracted at fixed time intervals, for example, one frame is extracted every 0.1 seconds. The number of frames to be extracted can also be determined based on the frame rate of the video, for example, 25 frames are extracted per second. The extracted frames are stored.

[0065] 2.2. Calculate the difference value: Content-based frame difference analysis determines the key frame by comparing the content difference between adjacent frames. First, the pixel difference between adjacent frames is calculated, and the absolute difference, square difference, etc. can be used.

[0066] 2.3. Set threshold: Set a threshold based on the difference value. If the difference exceeds the threshold, it is considered that a large change has occurred between the two frames, and one of the frames may be a key frame.

[0067] 2.4. Further screening: You can further screen key frames by combining other factors, such as time interval, motion amplitude, etc. For example, if the time interval between two frames is short but the difference is large, it may be caused by noise or fast motion and is not necessarily a key frame.

[0068] 3. Visual Feature Extraction

[0069] Use the corresponding feature extraction method to extract visual features from key frames, including color features, texture features, shape features, motion features, etc. Or use SI FT, SURF or deep learning models to extract and describe visual features in the frame.

[0070] 3.1. Extracting color features: The color features of key frames can be described by extracting color histograms, color moments, etc. The color histogram represents the distribution of different colors in the image and can be generated by counting the pixel values ​​of each color channel. The color moment describes the color features by calculating the mean, variance and other statistics of the image color.

[0071] 3.2. Extract texture features: Use gray-level co-occurrence matrix or local binary pattern method. Gray-level co-occurrence matrix describes texture features by calculating the gray value relationship of pixel pairs in the image. Local binary pattern describes texture features by comparing pixels with their surrounding pixels and generating binary codes.

[0072] 3.3. Extract shape features: edge detection, Hough transform, Fourier descriptor and other methods can be used. Edge detection can extract the contour of an object by detecting the edges in the image; Hough transform can be used to detect geometric shapes such as straight lines and circles in the image; Fourier descriptor describes the shape features by performing Fourier transform on the contour of the object.

[0073] 3.4. Extract motion features: Motion features can be extracted by calculating optical flow, motion vectors, etc. Optical flow refers to the speed and direction of movement of pixels in an image, which can be obtained by calculating the pixel displacement between adjacent frames. Motion vectors are obtained by comparing the position changes of objects in adjacent frames.

[0074] 3.5. SI FT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features) are commonly used feature extraction algorithms, which can extract local features in images and have scale invariance and rotation invariance.

[0075] 3.6. Models based on deep learning, such as convolutional neural networks (CNN), can also extract features. By training on large-scale image datasets, CNN can learn high-level feature representations of images.

[0076] These algorithms can automatically extract features from images and represent them in vector form for subsequent processing and analysis.

[0077] 4. Lens Type Feature Modeling

[0078] According to the definition of lens type, establish the feature model of each lens type. Analyze and learn the characteristics of different lens types, such as motion speed, scene depth, and the relationship between the subject and the background. Figure 4 As shown, it specifically includes the following contents:

[0079] 4.1. Determine the characteristics of the lens type: Identify different lens types, such as panoramic lens, medium shot, close-up shot, etc. For each lens type, determine its possible characteristics. For example, a panoramic lens may have characteristics such as a wide field of view, a large depth of scene, and a slow movement speed; a close-up shot may focus on a small subject, have a blurred background, and move relatively slowly.

[0080] 4.2. Establish feature model: By analyzing a large number of videos with known shot types, the typical features of each type are summarized and a feature model is established.

[0081] 4.3. Analyze and learn the characteristics of different shot types. The motion speed can be determined by calculating the displacement of objects between video frames. For example, the optical flow algorithm can be used to estimate the movement direction and speed of pixels, and then the average motion speed of the entire picture or a specific object can be calculated.

[0082] 4.4. Scene depth can be estimated by some computer vision techniques, such as stereo vision algorithms or depth estimation methods based on deep learning. Scene depth can also be roughly judged by observing the size and occlusion relationship of objects in the picture.

[0083] 4.5. The relationship between the subject and the background can be determined by image segmentation algorithm to determine the position and range of the subject, and then analyze the differences in color, texture, contrast and other features between the subject and the background.

[0084] 5. Lens type identification

[0085] The extracted features are analyzed using a trained classifier (such as support vector machine, random forest or neural network). The shot type is identified based on the degree of match between the features and the shot type feature model. Taking the trained SVM support vector machine classifier as an example, the shot type is determined based on the position of the feature vector in the hyperplane, thereby realizing automatic recognition of the video shot type. Figure 5 As shown, it specifically includes the following contents:

[0086] 5.1. Substitute the eigenvector extracted in the fourth step into the hyperplane equation to obtain the position of the hyperplane.

[0087] 5.2. The lens type is preliminarily determined based on the sign of the calculated value. If it is greater than or equal to 0, the feature vector is judged to be a positive class (representing a lens type). If the value is less than 0, it is judged to be a negative class (another lens type).

[0088] 5.3. Compare the extracted feature vector with the lens type feature model of the corresponding class for similarity, and use distance measurement methods, such as Euclidean distance, cosine distance, etc., to measure the similarity between the feature vector and each lens type feature model.

[0089] 5.4. Select the lens type with the shortest distance as the recognition result. You can also set a threshold. If the distance is less than the threshold, it is considered a successful match, otherwise it is considered that the lens type cannot be determined.

[0090] 6. Result Analysis and Optimization

[0091] Analyze the recognition results and evaluate the accuracy of recognition. According to the feedback of the recognition results, adjust the feature extraction algorithm or classifier parameters to optimize the recognition performance. Specifically, it includes the following contents:

[0092] 6.1. Compare the recognition results with the results of manual annotation, and calculate the accuracy, recall, F1 value and other indicators to evaluate the accuracy of recognition, etc. Analyze the cases of misaligned recognition and find out the possible reasons, such as inaccurate feature extraction, poor classifier performance, etc.

[0093] 6.2. If feature extraction is inaccurate, you can try to adjust the parameters of the feature extraction algorithm, such as the parameter settings of the SI FT and SURF algorithms, or try to use different feature extraction methods.

[0094] If the classifier performance is poor, you can adjust the classifier parameters, such as the kernel function parameters of SVM, the number of trees of random forest, etc. You can also try to use different classifiers or integrate multiple classifiers to improve performance.

[0095] In this embodiment, a video shot type recognition system based on the picture analysis principle is provided, and the system can implement the above-mentioned method. The system includes:

[0096] (1) Video input and preprocessing module: denoising, frame rate adjustment, and resolution standardization operations are performed on the input video files.

[0097] (2) Video frame extraction and key frame selection module: extract video frames from the preprocessed video files, and extract key frames from the video frames through a content-based frame difference analysis method.

[0098] (3) Visual feature extraction module: Use the corresponding feature extraction method to extract and describe the visual features of the key frames respectively; the visual features include color features, texture features, shape features and motion features.

[0099] (4) Shot type feature modeling module: According to the definition of shot type, a feature extraction model is established for each shot type based on the visual features of the key frames of the video file; and the feature vectors of different shot types are analyzed and learned.

[0100] (5) Lens type recognition module: The trained classifier is used to analyze the feature vector of the lens type, and the lens type is identified by calculating the matching degree between the feature vector of each lens type and the corresponding lens type feature model.

[0101] (6) Result analysis and optimization module: Analyze the recognition results, evaluate the recognition accuracy, and adjust the parameters of the feature extraction method and / or classifier based on the feedback of the recognition results, or directly replace the feature extraction method and / or classifier to optimize the recognition performance.

[0102] (7) User interface: used by users to upload video files, start the recognition process, and display the recognition results.

[0103] In this embodiment, the following measures are adopted to meet the real-time processing requirements of the present invention, including reducing the computational complexity and improving the processing speed: Implementing multi-threading or distributed processing to improve the parallelism of video processing.

[0104] (1) Reducing computational complexity: This can be achieved by simplifying the algorithm, reducing feature dimensions, and adopting efficient data structures. For example, methods such as principal component analysis (PCA) can be used to reduce the dimension of feature vectors and reduce the amount of computation.

[0105] (2) Improving processing speed: This can be achieved by optimizing the code, using parallel computing, hardware acceleration, and other methods. For example, using multithreading or distributed processing to process multiple video frames simultaneously can improve processing speed.

[0106] (3) Multithreaded processing can divide the video processing task into multiple subtasks, each of which is executed in an independent thread. This can fully utilize the performance of multi-core processors and improve processing speed.

[0107] (4) Distributed processing: Using distributed computing frameworks such as Hadoop and Spark, video processing tasks can be distributed to multiple computing nodes for parallel processing.

[0108] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0109] The present invention provides a method and system for identifying video shot types based on the principle of picture analysis. The present invention can accurately identify different types of shots through multiple feature extractions and advanced classifiers, greatly reducing the possibility of misidentification; the optimization algorithm meets the real-time processing requirements, reduces the computational complexity, and improves the processing speed. The present invention analyzes and learns different types of features, establishes a feature model for each shot type, so that the present invention can adapt to different types of video content and shooting styles, and provides good versatility and adaptability. The user interface developed by the present invention can allow users to upload videos, and the interactive functions provided can allow users to better participate in and adjust the recognition process, thereby improving the user experience.

[0110] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A video shot type recognition method based on picture analysis principle, characterized in that: The following steps are included: S1. Video frame extraction and key frame selection: extract video frames from the preprocessed video files, and extract key frames from the video frames through a content-based frame difference analysis method; S2, visual feature extraction: using corresponding feature extraction methods to extract and describe the visual features of key frames respectively; Visual features include color features, texture features, shape features, and motion features; S3, shot type feature modeling: according to the definition of shot type, based on the visual features of the key frames of the video file, a feature extraction model is established for each shot type; And analyze and learn the feature vectors of different lens types; S4. Lens type identification: Analyze the feature vector of the lens type using the trained classifier, and identify the lens type by calculating the matching degree between the feature vector of each lens type and the corresponding lens type feature model.

2. The video shot type identification method based on picture analysis principle according to claim 1 is characterized in that: Step S1 specifically includes the following contents: S11, extracting video frames from the video file at fixed time intervals, or determining the number of extracted video frames according to the frame rate of the video file; and storing the extracted video frames; S12, the content-based frame difference analysis method determines the key frame in the video frame by comparing the content difference between adjacent video frames, and if the content difference value between adjacent video frames is greater than a preset threshold, it is determined that there is a key frame in the adjacent video frame; S13, further filter the key frames based on the time interval and the motion amplitude.

3. The video shot type identification method based on picture analysis principle according to claim 1 is characterized in that: Step S2 specifically includes the following contents: S21, extracting and describing the color features of the key frame by extracting a color histogram and a color moment; S22, extracting and describing texture features of the key frame using a gray level co-occurrence matrix or a local binary pattern method; S23, using edge detection, Hough transform, and Fourier descriptor methods to extract and describe the shape features of the key frames; S24. Extract and describe the motion features of the key frames by calculating optical flow and motion vectors.

4. The video shot type identification method based on picture analysis principle according to claim 1 is characterized in that: Step S3 specifically includes the following contents: S31. Identify different types of lenses and, for each lens type, determine the characteristics it may have; S32, analyzing the visual features of key frames of a large number of video files of known shot types, summarizing the typical features of each shot type, and establishing a feature model for each shot type; S33. Analyze and learn the feature vectors of different shot types, including motion speed, scene depth, and the relationship between the subject and the background.

5. The video shot type identification method based on picture analysis principle according to claim 1 is characterized in that: Step S4 specifically includes the following contents: S41, substituting the feature vector extracted in step S3 into the hyperplane equation to obtain the position of the hyperplane; S42, preliminarily determining the lens type according to the calculated value, if the value is greater than or equal to 0, the feature vector is determined to be a positive class, otherwise it is determined to be a negative class; S43, performing a similarity comparison between the extracted feature vector and the lens type feature model of the corresponding class to measure the similarity between the feature vector and each lens type feature model; S44, selecting the lens type with the highest similarity as the recognition result.

6. The video shot type identification method based on picture analysis principle according to claim 1 is characterized in that: Step S4 also includes: S5. Result analysis and optimization: Analyze the recognition results, evaluate the recognition accuracy, and adjust the parameters of the feature extraction method and / or classifier based on the feedback of the recognition results, or directly replace the feature extraction method and / or classifier to optimize the recognition performance.

7. The video shot type identification method based on picture analysis principle according to claim 6 is characterized in that: Step S5 specifically includes the following contents: S51, comparing the recognition result with the result of manual annotation, and using multiple evaluation indicators to evaluate the accuracy of recognition; analyzing the case of misaligned recognition and determining the cause of misaligned recognition; S52. If the reason for misalignment recognition is inaccurate feature extraction, adjust the parameters of the feature extraction algorithm, or try to use other feature extraction algorithms; if the reason for misalignment recognition is poor classifier performance, adjust the parameters of the classifier, or try to use other classifiers or integrate multiple classifiers.

8. The video shot type identification method based on picture analysis principle according to claim 1 is characterized in that: Before step S1, the method further includes: S0, video input and preprocessing: denoising, frame rate adjustment, and resolution standardization operations are performed on the input video files.

9. The video shot type identification method based on picture analysis principle according to claim 8 is characterized in that: Step S0 specifically includes the following contents: S01, receiving video files of various formats as input data; S02. Use median filtering and Gaussian filtering to effectively remove noise interference in video files; S03, adjusting the frame rate of the video file to an appropriate value according to subsequent processing requirements and hardware performance; S04. Standardize the resolution of the video files to unify the resolution of each video file.

10. A video shot type recognition system based on picture analysis principle, characterized in that: The system can implement the method described in any one of claims 1 to 9, and the system includes: Video input and preprocessing module: denoise, adjust frame rate, and standardize resolution of input video files Video frame extraction and key frame selection module: extracts video frames from the preprocessed video files, and extracts key frames from the video frames through content-based frame difference analysis method; Visual feature extraction module: Use corresponding feature extraction methods to extract and describe the visual features of key frames respectively; Visual features include color features, texture features, shape features, and motion features; Shot type feature modeling module: According to the definition of shot type, based on the visual features of the key frames of the video file, a feature extraction model is established for each shot type; and the feature vectors of different shot types are analyzed and learned; Lens type identification module: Use the trained classifier to analyze the feature vector of the lens type, and identify the lens type by calculating the matching degree between the feature vector of each lens type and the corresponding lens type feature model. Result analysis and optimization module: analyzes the recognition results, evaluates the recognition accuracy, and adjusts the parameters of the feature extraction method and / or classifier based on the feedback of the recognition results, or directly replaces the feature extraction method and / or classifier to optimize the recognition performance. User interface: used for users to upload video files, start the recognition process, and display the recognition results.

Images