Video cutting method and system based on thermodynamic diagram and mask
By supporting the combination of vector machine classification and thermal diagram, and mask technology, intelligent analysis of the content of the video frame content solves the problem of poor video cutting effects in existing technology, and achieves high -quality video cutting effects.
Patent Information
- Application Number
- CN202411367118.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-01-21
AI Technical Summary
The prior art lacks targeted processing strategies when processing different types of video content, resulting in unsatisfactory video cropping effects.
The support vector (SVM) is used to classify the video frame, and combined with thermal diagram analysis and mask processing technology to achieve intelligent video cutting. Specific steps include classification of video frames according to four types of scenes, calculating thermal diagrams and masks of various scenarios, selected areas with the largest multiplied value as the center of the cutting area, and filtering the time domain and airspace.
The high -quality cutting of the video has been achieved, the core visual information and narrative focus of the video retains the optimization and content quality of the cutting effect.
Smart Images
Figure CN119342280A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video processing technology, and in particular to a method for realizing intelligent video cropping by using a support vector machine (SVM) to classify video frames and combining heat map analysis and mask processing technology. Background Art
[0002] As video content becomes increasingly rich, efficient and accurate video cropping to highlight the core content has become an important requirement for video editing and processing. Existing technologies often lack targeted processing strategies when processing different types of video content (such as fixed shots, shot switching, shot following, and large movements), resulting in unsatisfactory cropping effects. Therefore, it is particularly important to develop a technology that can intelligently identify video content types and optimize cropping strategies accordingly.
[0003] Support Vector Machine (SVM) is a commonly used classification algorithm. Its basic idea is to map data into a high-dimensional feature space and find an optimal hyperplane in the space so that samples of different categories can be separated as much as possible. SVM determines the optimal hyperplane by finding support vectors (sample points on the hyperplane) and uses these support vectors for classification. It performs well in high-dimensional feature spaces. SVM can still maintain good classification performance when processing a small number of support vectors and has high efficiency for large-scale data sets.
[0004] Heatmaps can intuitively show the differences in data through changes in color depth. They are statistical charts that display data by coloring blocks of color. When drawing, you need to specify the color mapping rules. For example, larger values are represented by darker colors, smaller values by lighter colors; larger values are represented by warmer colors, smaller values by cooler colors, and so on. Summary of the invention
[0005] This paper proposes a video cropping method based on heat map and mask, which aims to achieve high-quality video cropping by intelligently analyzing the content characteristics of video frames and adopting a customized processing flow. This method innovatively combines support vector machine classification, heat map generation and mask technology. The specific steps are as follows:
[0006] A video cropping method based on a heat map and a mask comprises the following steps:
[0007] S1: Classify video frames into four types of scenes: fixed lens, lens switching, lens following, and large motion;
[0008] S2: Calculate the heat map of the video frame images of each scene classification, and calculate the mask of the video frame images of the scene with fixed lens, following lens, and large motion;
[0009] S3: For scenes with fixed lens, lens following, and large motion, the heat map and the mask are multiplied, and the area with the largest product value is selected as the center of the cropping area, and the area is cropped according to the size of the cropping area, and the cropped area is filtered in the time and space domains. For scenes with lens switching, the maximum value of the two-dimensional matrix in the heat map is used as the center of the cropping area, and the area is cropped according to the size of the cropping area, and the cropped area is filtered in the time and space domains.
[0010] Preferably, the classification of the video frames in step S1 specifically includes the following steps:
[0011] S1.1: The video frame is first converted to grayscale, and the frame difference or optical flow intensity of adjacent input frames is calculated; the video frame difference or optical flow intensity is compressed into a 1*w-dimensional array in the vertical direction, and then the 1*w-dimensional array is compressed horizontally into a 1*N-dimensional vector, where w is the width of the video frame and N is the number of blocks after horizontal compression, which means that after merging several adjacent columns of the image, the entire image is divided into N independent blocks in the horizontal direction;
[0012] S1.2: The obtained 1*N dimensional vector is used as the input of the SVM classifier, and the SVM is used to determine the category to which the video frame belongs.
[0013] Preferably, the calculation steps of the heat map include: extracting a single frame image from the video, using the single frame image as the input of the deep neural network yolov5 model, and extracting the last layer of network feature map as the heat map.
[0014] Preferably, the steps for calculating the mask for a scene with a fixed lens and large motion include: extracting a single frame image from the video, graying the single frame image, calculating the grayscale difference between adjacent frame images pixel by pixel and taking the absolute value to form a two-dimensional matrix as a mask; or calculating the optical flow of adjacent frames, and taking the modulus of the optical flow vector pixel by pixel as a mask.
[0015] Preferably, the mask calculation step of the lens following scene includes: extracting a single frame image from the video, using the single frame image horizontally with a middle position value of 2 and a value of 1 at both edges, and performing linear interpolation in the transition area to obtain a mask of an area with a larger middle position and a gradually smaller area on both sides.
[0016] Preferably, the specific steps of performing time domain and spatial domain filtering on the cropped area include: first, determining the coordinate difference between the cropped area, i.e., the target area of the current frame, and the target area of the previous frame; if the difference is small, determining that the target has slightly changed position; further determining the difference between the two thermal maps before and after; if the difference is large, indicating that the object has moved laterally, and therefore using the coordinates of the current target area for cropping; if the difference is small, indicating that the object is just shaking in place, and using the coordinates of the target area of the previous frame for cropping; if the coordinate difference between the target area of the current frame and the target area of the previous frame is large, determining whether the current video frame is a switching video frame; if so, using the median of the target area of the last five frames for cropping; if not, determining whether the coordinates of the previous frame will appear in several future frames; if so, using the coordinates of the previous frame for cropping; if not, using the median of the target area of the last five frames for cropping.
[0017] The present invention also provides a video cropping system based on a thermal map and a mask, comprising:
[0018] The scene classification module is used to classify video frames into four types of scenes: fixed shot, shot switching, shot following, and large motion;
[0019] The heat map and mask calculation module is used to calculate the heat map of the video frame images of each scene classification, and calculate the mask of the video frame images of the scene with fixed lens, lens following, and large motion;
[0020] The video cropping module multiplies the heat map and the mask for scenes with fixed lens, lens following, and large motion, selects the area with the largest product value as the center of the cropping area, and crops according to the size of the cropping area; for scenes with lens switching, the maximum value of the two-dimensional matrix in the heat map is used as the center of the cropping area, and crops according to the size of the cropping area;
[0021] The time domain and space domain filtering module is used to perform time domain and space domain filtering on the cropped area.
[0022] Beneficial effects: The present invention aims to achieve high-quality video cropping by intelligently analyzing the content characteristics of video frames and adopting a customized processing flow. The method innovatively combines support vector machine classification, heat map generation and masking technology to jointly ensure the optimization of video cropping effects and content quality. The invention is expected to be widely used in the fields of media, film and television, sports live broadcast, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a general flow chart of a specific embodiment of the present invention;
[0024] Figure 2 It is a detailed implementation diagram of a specific embodiment of the present invention. DETAILED DESCRIPTION
[0025] This paper proposes a video cropping method based on heat map and mask, which aims to achieve high-quality video cropping by intelligently analyzing the content characteristics of video frames and adopting a customized processing flow. This method innovatively combines support vector machine classification, heat map generation and mask technology. The specific steps are as follows:
[0026] S1: Using support vector machine (SVM) technology, video frames are classified into four categories: fixed lens, lens switching, lens following, and large motion. This classification process is based on SVM classification technology to ensure the accuracy of classification;
[0027] S2: Fixed-lens scenario: Calculate heatmaps and extract motion intensity as masks to protect key areas that remain essentially unchanged in the video;
[0028] S3: Shot switching scene: Directly calculate the heat map as the main basis for video cropping, especially at the moment of shot switching;
[0029] S4: Camera following scene: After calculating the heat map, combined with the regional mask (the value in the center of the image is the largest and the value gradually decreases towards the sides), the middle area in outdoor and sports videos is strengthened, and the key area is moved smoothly to maintain fluency;
[0030] S5: Large motion scenes: Calculate the heat map and extract the motion intensity as a mask to ensure smooth transition of the motion area and accurate tracking of key areas.
[0031] S6: Cropping decision: Based on the mask and heat map obtained after the above processing, the two are multiplied, and the distribution of their product values is counted. The peak value is selected as the center of the cropping area, and the cropping is performed according to the size of the cropping area to achieve accurate cropping of the video content and retain the most important visual information. On this basis, temporal / spatial filtering is performed to further improve the image stability.
[0032] Furthermore, the classification of the video frames in step S1 specifically includes the following steps:
[0033] S1.1: The video frames read into the program are first converted to grayscale, and the frame difference between adjacent input frames is calculated. Considering that the dimensions of video frames of different sizes are also different, they are unified. Assuming that the width and height of the input video frame are w and h, the frame difference is first compressed into a 1*w-dimensional array, and then the 1*w-dimensional array is processed into a 1*N-dimensional vector with the same dimension.
[0034] S1.2: The 1*N dimensional vector obtained above is used as the input of the SVM (support vector machine) classifier, and the SVM is used to determine the category to which the video frame belongs.
[0035] Furthermore, the processing of the fixed lens scene in step S2 specifically includes the following steps:
[0036] S2.1: For scenes with fixed lenses, the characteristic indicators of such scenes focus on reflecting low motion energy, that is, most pixels change slightly. The processing methods adopted are: 1. Calculate the heat map: extract a single frame image from the video, use the single frame image as the input of the deep neural network yolov5 model, and extract the last layer of network feature map as the heat map; 2. Extract motion intensity as a mask: extract a single frame image from the video, grayscale the single frame image, calculate the grayscale difference of adjacent frame images pixel by pixel and take the absolute value to form a two-dimensional matrix as a mask; or calculate the optical flow of adjacent frames, take the modulus of the optical flow vector pixel by pixel as a mask to enhance the key areas in the video.
[0037] Furthermore, the step S3 specifically includes the following steps for the shot switching scene:
[0038] S3.1: For the scene of lens switching, pay attention to the significant increase in pixel mutation rate, which usually means the rapid conversion of scenes. Use the deep neural network yolov5 model, input the current frame image, and extract the last layer of network feature map as the heat map, as the main basis for video cropping, take the maximum value in the two-dimensional matrix in the heat map as the center of the cropping area, and crop according to the size of the cropping area. This step only uses the heat map of the frame of the screen transition as the basis for cropping, ensuring a natural transition and avoiding the influence of the residual information of the previous video image on the cropping.
[0039] Furthermore, the step S4 specifically includes the following steps for the lens following scene:
[0040] S4.1: For the camera follow scene, the emphasis is on the trend of continuous motion, which is reflected in the consistency and coherence of the motion vector. The heat map calculation steps include: using the deep neural network yolov5 model, inputting the current frame image, and extracting the last layer of network feature map as the heat map; the mask calculation steps include: extracting a single frame image from the video, setting the mask value of the horizontal middle position of the single frame image to 2, and the mask value at the left and right edges of the image to 1, and taking linear interpolation in the transition area to obtain a mask with a large middle area and gradually decreasing on both sides, strengthening the middle area in outdoor and sports videos, and smoothly moving the key area to maintain fluency.
[0041] Furthermore, the step S5 specifically includes the following steps for scenes with large movements:
[0042] S5.1: Large movements are manifested as high movement amplitudes and complex movement patterns. It is necessary to comprehensively consider the diversity of movement intensity and direction changes. Therefore, the heat map is calculated as in S2.1, and the motion intensity matrix is extracted as a mask through the frame difference method or optical flow method. For the case where the target moves back and forth, the smooth transition of the moving area and the accurate tracking of the key area are ensured.
[0043] Furthermore, the cropping decision in step S6 specifically includes the following steps:
[0044] S6.1: Comprehensively analyze the weight mask and heat map obtained after processing each frame, multiply the weight mask with the heat map matrix, and count the distribution of their product values. Select the peak value as the center of the cropping area and crop according to the size of the cropping area. This strategy ensures that the cropping process can not only remove redundant borders, but also maximize the retention of the core visual information and narrative focus of the video, thereby achieving high-quality content cropping.
[0045] S6.2: The temporal / spatial filtering technology refers to that the center coordinates of the video cropping remain unchanged in the time dimension according to the coordinate keeping algorithm, thereby enhancing the stability of the algorithm. The specific implementation rules are as follows: First, determine the coordinate difference between the target area of the current frame and the target area of the previous frame. If the difference is small, it can be determined that the target has changed its position slightly. At this time, the difference between the two thermal maps before and after can be further determined. If the difference is large, it means that the object has moved horizontally, so the coordinates of the current target area can be used for cropping. If the difference is small, it means that the object is just shaking in place. In order to maintain a good visual effect and avoid continuous image shaking, the coordinates of the target area of the previous frame are used for cropping. If the coordinate difference between the target area of the current frame and the target area of the previous frame is large, determine whether the current video frame is a switching video frame. If so, use the median of the target area of the next 5 frames for cropping. If not, determine whether the coordinates of the previous frame will appear in the next few frames. If so, it means that the screen flickers and switches. Here, the coordinates of the previous frame are used for cropping. If not, it means that the target in the screen has switched positions, and the median of the target area of the next 5 frames is used for cropping. This rule effectively ensures the accuracy of the cropping area and avoids jitter and flickering of the image.
Claims
1. A video cropping method based on heat map and mask, characterized in that: The steps include: S1: Classify video frames into four types of scenes: fixed lens, lens switching, lens following, and large motion; S2: Calculate the heat map of the video frame images of each scene classification, and calculate the mask of the video frame images of the scene with fixed lens, following lens, and large motion; S3: For scenes with fixed lens, lens following, and large motion, the heat map and the mask are multiplied, and the area with the largest product value is selected as the center of the cropping area, and the area is cropped according to the size of the cropping area, and the cropped area is filtered in the time and space domains. For scenes with lens switching, the maximum value of the two-dimensional matrix in the heat map is used as the center of the cropping area, and the area is cropped according to the size of the cropping area, and the cropped area is filtered in the time and space domains.
2. The video cropping method based on heat map and mask according to claim 1, characterized in that: The classification of the video frames in step S1 specifically includes the following steps: S1.1: The video frame is first converted to grayscale, and the frame difference or optical flow intensity of adjacent input frames is calculated; the video frame difference or optical flow intensity is compressed into a 1*w-dimensional array in the vertical direction, and then the 1*w-dimensional array is compressed horizontally into a 1*N-dimensional vector, where w is the width of the video frame and N is the number of blocks after horizontal compression, which means that after merging several adjacent columns of the image, the entire image is divided into N independent blocks in the horizontal direction; S1.2: The obtained 1*N dimensional vector is used as the input of the SVM classifier, and the SVM is used to determine the category to which the video frame belongs.
3. The video cropping method based on heat map and mask according to claim 1, characterized in that: The calculation steps of the heat map include: extracting a single frame image from the video, using the single frame image as the input of the deep neural network yolov5 model, and extracting the last layer of network feature map as the heat map.
4. The video cropping method based on heat map and mask according to claim 1, characterized in that: The calculation steps of the mask for scenes with fixed shots and large movements include: extracting a single frame image from the video, graying the single frame image, calculating the grayscale difference between adjacent frame images pixel by pixel and taking the absolute value to form a two-dimensional matrix as a mask; or calculating the optical flow of adjacent frames, taking the modulus of the optical flow vector pixel by pixel as a mask.
5. The video cropping method based on heat map and mask according to claim 1, characterized in that: The mask calculation steps of the lens following scene include: extracting a single frame image from the video, using the single frame image horizontal middle position value as 2, the value at the two side edges as 1, and linear interpolation in the transition area to obtain a mask of the area with a larger middle and gradually smaller sides.
6. The video cropping method based on heat map and mask according to claim 1, characterized in that: The specific steps of performing temporal and spatial filtering on the cropped area include: first, determining the coordinate difference between the cropped area, i.e., the target area of the current frame, and the target area of the previous frame; if the difference is small, it is determined that the target has slightly changed its position; and further determining the difference between the two thermal maps before and after; if the difference is large, it indicates that the object has moved laterally, and therefore, the current target area coordinates are used for cropping; if the difference is small, it indicates that the object is just shaking in place, and the target area coordinates of the previous frame are used for cropping; if the coordinate difference between the target area of the current frame and the target area of the previous frame is large, determining whether the current video frame is a switching video frame; if so, using the median of the target area of the next five frames for cropping; if not, determining whether the coordinates of the previous frame will appear in several future frames; if so, using the coordinates of the previous frame for cropping; and if not, using the median of the target area of the next five frames for cropping.
7. A video cropping system based on heat map and mask, characterized in that: include: The scene classification module is used to classify video frames into four types of scenes: fixed shot, shot switching, shot following, and large motion; The heat map and mask calculation module is used to calculate the heat map of the video frame images of each scene classification, and calculate the mask of the video frame images of the scene with fixed lens, lens following, and large motion; The video cropping module multiplies the heat map and the mask for scenes with fixed lens, lens following, and large motion, selects the area with the largest product value as the center of the cropping area, and crops according to the size of the cropping area; for scenes with lens switching, the maximum value of the two-dimensional matrix in the heat map is used as the center of the cropping area, and crops according to the size of the cropping area; The time domain and space domain filtering module is used to perform time domain and space domain filtering on the cropped area.