A pipeline action video key frame extraction method based on optical flow estimation and ORB

Abstract

The present invention relates to the field of key frame extraction of pipeline action videos, and relates to a key frame extraction method for pipeline action videos based on optical flow estimation and ORB. The method comprises the following steps: inputting a pipeline action video; calculating an optical flow difference value; searching for key frames according to significant changes in the optical flow difference value; clustering according to the optical flow difference value; extracting local features of each image frame using an ORB algorithm; converting each image frame into a feature vector according to the local features of the image frame extracted by the ORB algorithm; grouping image frames of the same category according to numbers; calculating the information entropy of each image frame; clustering according to the feature vectors; and composing a new video from all key frames in ascending order of numbers. The method can use different key frame extraction strategies for different action segments according to the magnitude of the action amplitude in the pipeline action video, and the extracted key frames have strong action coherence and logic, so as to more completely present the operation process of pipeline production while ensuring the compression rate.

Description

Technical Field

[0001] The present invention relates to the field of pipeline action video key frame extraction, and in particular to a pipeline action video key frame extraction method based on optical flow estimation and ORB. Background Art

[0002] In industrial production, many assembly line operations still rely on manual labor. The primary drawback of this production method is low production efficiency. Numerous factors contribute to this inefficiency, but substandard worker performance during the production process is a core factor. Therefore, scientifically regulating and managing worker operations is essential.

[0003] Currently, many factories use cameras to record the entire workflow of assembly line workers and then analyze the video content to manage their production operations. However, in production line action videos, workers' operations are a repetitive process, and the information redundancy of such videos is high. Directly processing such videos is not only inefficient but also wastes computing and storage resources. By first extracting keyframes from such videos and then processing the extracted keyframes, we can reduce computer resource consumption and improve processing efficiency.

[0004] Moreover, existing key frame extraction technologies, such as motion analysis, shot boundary method, and feature clustering method, do not take into account the uniqueness of assembly line action videos. They often only extract frames with relatively large information entropy from the video as key frames, and adopt the same key frame extraction strategy for all action clips in the video. They do not consider the continuity and logic of the actions between key frames. When the existing key frame extraction technology is used to process the assembly line action video, the extracted key frames are sequentially combined into a new video. This video cannot well show the operation process of the assembly line workers. Summary of the Invention

[0005] The purpose of the present invention is to provide a key frame extraction method for pipeline action videos based on optical flow estimation and ORB, which can use different key frame extraction strategies for different action segments according to the size of the action amplitude in the pipeline action video. The action continuity and logic between the extracted key frames are strong, and the operation process of pipeline production can be more completely displayed while ensuring the compression rate.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A pipeline action video key frame extraction method based on optical flow estimation and ORB includes the following steps:

[0008] S1, input pipeline action video and grayscale the video;

[0009] S2. Calculate the optical flow difference value. Starting from the second frame, use the DIS algorithm to generate an optical flow vector for each pixel between the previous frame and the current frame. Then calculate the optical flow difference value between the current frame and the previous frame. The optical flow difference value of the first frame is the same as that of the second frame by default. In this way, each frame corresponds to an optical flow difference value. At this time, each image frame has two attributes: number and optical flow difference value.

[0010] S3. Find a key frame based on a significant change in the optical flow difference value. Traverse each image frame except the first and last frames. If the absolute value of the difference between the optical flow difference value of the current image frame and the optical flow difference value of the next frame is greater than or equal to r, then the current image frame is considered a key frame. The first and last frames are considered key frames by default.

[0011] S4. Clustering based on optical flow difference values: Perform k-means clustering based on the optical flow difference values ​​of each frame, clustering into x classes, calculating the proportion of the class center value of each class to the sum of all class center values, and using a dictionary to store the class number and the proportion of its class center value. At this point, each image frame has three attributes: number, category, and optical flow difference value;

[0012] S5. Extract local features of each image frame using the ORB algorithm;

[0013] S6. Convert each image frame into a feature vector based on the local features of the image frame extracted by the ORB algorithm. At this point, each image frame has four attributes: number, category, feature vector, and optical flow difference value;

[0014] S7. Group image frames of the same category according to their numbers: Arrange image frames of the same category in ascending order of numbers, traverse each frame, and the first frame of each category is defaulted to the group with group number 0 in the category. Subtract the number value of the previous frame from the number value of the current frame. If the difference is less than or equal to m, the current frame and the previous frame are classified into the same group. Otherwise, the group number is increased by one, and the current frame is classified into another group. At this point, each image frame has five attributes: number, category, group, feature vector, and optical flow difference value;

[0015] S8. Calculate the information entropy of each image frame. At this point, each image frame has six attributes: number, category, group, feature vector, information entropy, and optical flow difference value.

[0016] S9. Clustering based on eigenvectors: Perform k-means clustering on each group based on the eigenvectors. The number of clusters for each group is rounded up, and the frame with the largest information entropy is selected as the key frame. The number of clusters for each group is: the current group length / (a ​​+ the proportion of the center value of the class to which the current group belongs * b);

[0017] S10: Merge the key frame sets obtained in steps S3 and S9, and compose a new video from all key frames in ascending order of number.

[0018] A further technical solution is that, in step S1, the image is converted from the BGR color space to a grayscale image using the cv2.cvtColor function in the OpenCV library.

[0019] A further technical solution is to use a dictionary to store the attributes of the frame in step S2, with the number as the key and the dictionary {"flow_diff": optical flow difference value} as the value. The specific process of calculating the optical flow difference value is:

[0020] Step 1. Use the DIS algorithm to generate an optical flow vector for each pixel between the previous frame and the current frame. The optical flow vector describes the movement of these points between the two frames. The optical flow vector can be expressed as (u, v), where u is the horizontal pixel displacement and v is the vertical pixel displacement. Use the cv2.DISOpticalFlow_create function in the OpenCV library to calculate the optical flow vector.

[0021] The DIS algorithm process includes a given template patch in the reference image, and DIS finds the best matching subwindow in the query image by gradient descent. The goal of this process is to find a transformation vector u , so that the sum of the squared differences between the template and query positions in the subwindow is minimized, the expression is:

[0022] ,in T is the template patch in the reference image, I t+1 is the query image, x is the pixel position in the patch, u is the best matching transformation vector obtained;

[0023] DIS uses a multi-scale approach to gradually optimize patches from coarse to fine. At each scale, the displacement of the patch is initialized by the displacement of the previous scale, and then optimized. The displacement of each scale is combined into a dense displacement field through weighted averaging. The mathematical expression is:

[0024] ,in, U s ( x ) is each pixel x The dense displacement field ，N s is the number of patches at that scale, d i ( x ) is the patch and the deformed image in pixelsx The intensity difference at u i It's a patch i The estimated displacement, λ ix is an indicator function that indicates that the patch i Whether with location x overlapping;

[0025] DIS uses variational refinement to further improve the accuracy of the displacement field. This step is achieved by minimizing an energy function that includes a data term and a smoothing term. The mathematical expression of the energy function is:

[0026] ,in, E ( U ) is the energy to be minimized, Ψ is a robust penalty function, E I 、 E G 、 E S They are intensity data term, gradient data term and smoothing term;

[0027] Through the above steps, a dense optical flow field describing the displacement of each pixel from one frame to another is finally generated, and the optical flow vector of each pixel between the previous frame and the current frame is obtained from the generated dense optical flow field ( u , v );

[0028] Step 2: Calculate the modulus of each optical flow vector, expressed as:

[0029] ;

[0030] Step 3: Calculate the average value of the optical flow vector modulus, that is, add up the modulus of the optical flow vectors of all pixels and divide it by the total number of pixels. This value is used as the optical flow difference value of the current frame, expressed as:

[0031] , use a dictionary to store the attributes of the frame. The elements of the dictionary are also dictionaries, with the number as the key and the dictionary {"flow_diff": optical flow difference value} as the value.

[0032] A further technical solution is that in step S4, if the center value of a certain category is much larger than the center value of other categories, its center value is replaced by the second largest center value to prevent the center value of this category from having too high a proportion and making the center value proportions of other categories too low.

[0033] A further technical solution is that in step S5, the specific process of using the ORB algorithm to extract local features is as follows:

[0034] Step 1: Construct a scale space pyramid. Specifically, set a scale factor scaleFactor and the number of pyramid levels nlevels, and reduce each frame image into nlevels images according to the scale factor.

[0035] Step 2, FAST key point detection, specifically for each pixel point p in the nlevels image of the scale space pyramid of each frame image, with pixel p as the center, select 16 pixels on a circle with a radius of 3, if there are N consecutive points with brightness greater than I on the 16 pixels on the circle, p +T or less than I p -T, then pixel p can be considered as a feature point (N=9), and all feature points are used as FAST feature points of the current frame;

[0036] Step 3: Orient the key points by using the intensity centroid. Specifically, the ORB algorithm uses the vector starting from the key point and ending at the intensity centroid as the direction of the key point. For each key point, its direction θ is calculated as follows: ;

[0037] This formula defines a moment, through which the intensity centroid can be found, where I(x,y) is the pixel intensity value at coordinate (x,y), p and q are the order of the moment, and the centroid C Obtained by the following formula:

[0038] ;

[0039] in, m 00 Represents the sum of all pixel intensity values ​​in the image; m 10 is the sum of the products of the x-coordinates of all pixels in the image and their intensity values, m 01 It is the sum of the products of the y-coordinates of all pixels in the image and their intensity values;

[0040] The orientation θ of a keypoint is determined by the angle of the vector from the keypoint to its centroid:

[0041] ;

[0042] Step 4: Generate a rotation-invariant BRIEF descriptor. Specifically, the image is smoothed by Gaussian filtering, and then an S×S neighborhood (S=31) is taken around the key point. In the neighborhood, n point pairs (p,q) are selected by a selected method, and the point pairs are numbered starting from 1 in the order of their generation. The number is set to i, and the coordinate system of the key point is rotated according to the direction. θThe coordinates of the pixel pair (p′, q′) after rotation are calculated using the following rotation matrix:

[0043] ;

[0044] The pixel pair (p′, q′) inherits the number value i of the corresponding point pair (p, q). For each adjusted pixel pair (p′, q′), their intensity values ​​are compared. If I(p′)>I(q′), the i-th bit of the descriptor is set to 1, otherwise, the i-th bit of the descriptor is set to 0, and finally an n-bit binary string is generated.

[0045] A further technical solution is that in step S6, the specific process of using the BOF algorithm to convert each frame of image into a feature vector is as follows:

[0046] Step 1: Construct visual vocabulary, specifically set F={f1, f2,…, f N} is the set of all extracted ORB descriptors, feature clustering: use the k-means clustering algorithm to cluster F into K clusters. The k-means goal is to minimize the sum of the squares of the distances from each sample point to its cluster center, expressed as:

[0047] ;

[0048] The center c of each class j Represents a visual word, the set of cluster centers {c1,c2,…,c K} constitutes a vocabulary dictionary; Step 2, feature vectorization, specifically the ORB feature descriptor set F for each image frame I Each descriptor f in i , calculate its distance to all visual words, and then classify it into the nearest visual word, expressed as:

[0049] ;

[0050] A K-dimensional feature vector h is constructed for each image frame, and the j-th dimension represents the ORB feature descriptor of the image frame classified into the visual vocabulary c j The frequency of the ORB feature descriptor f of the image frame i Assigned to the visual word c j , then the value of the eigenvector in the jth dimension increases by 1, let h j Represents the value of the j-th dimension of the histogram, expressed as:

[0051] ;

[0052] Among them, l() is the indicator function;

[0053] Step 3: Normalize the feature vector. Specifically, use the L2 norm to make the modulus of the feature vector 1, which is expressed as:

[0054] ;

[0055] .

[0056] A further technical solution is that in step S8, the specific process of calculating information entropy is:

[0057] Step 1: Histogram calculation: Assume that the image frame has N pixels and the grayscale range is 0 to L−1. Then the histogram H(i) of the image represents the number of pixels with grayscale level i in the image.

[0058] Step 2: Calculate the probability distribution. Specifically, calculate the probability distribution based on the histogram, that is, the probability P(i) of each gray level appearing. Divide each value in the histogram by the total number of pixels N, and express it as:

[0059] ;

[0060] Among them, P(i) represents the gray level in the image i The probability of the pixel appearing;

[0061] Step 3: Calculate information entropy, expressed as:

[0062] .

[0063] A further technical solution is that in step S9, parameter a is used to ensure the basic compression rate, and parameter b determines the difference in compression rates of image frames of different categories. The larger b is, the greater the difference in compression rate between categories with large optical flow difference values ​​and categories with small optical flow difference values, and vice versa.

[0064] A further technical solution is that the k-means clustering in steps S4, S6 and S9 can be performed using the Kmeans function of the scikit-learn library. The specific process is as follows:

[0065] Step 1: Determine the number of clusters and initialize the cluster centers, which can be divided into:

[0066] Step A: Set the number of clusters to k and randomly select a sample as the first cluster center;

[0067] Step B: Calculate the Euclidean distance from each sample point x to the nearest cluster center, denoted as D(x). In n-dimensional space, the Euclidean distance between two points P1(x11, x12,…,x1n) and P2(x21, x22,…,x2n) is calculated as:

[0068] ;

[0069] Step C: Based on the distance calculated in step B, calculate a selection probability for each sample point x that is not selected as the cluster center, expressed as:

[0070] ;

[0071] Step D, repeat steps B and C until k cluster centers are selected;

[0072] Step 2: Assign the sample to the nearest cluster center: Calculate the Euclidean distance from each sample to each cluster center, and then assign the sample to the cluster represented by the nearest cluster center;

[0073] Step 3: Update the cluster center: After all samples are assigned to clusters, update the cluster center of each cluster. The new cluster center is the average value of the coordinates of the sample points it contains.

[0074] Step 4: Repeat steps 2 and 3 until the convergence condition is met: the change in cluster center is less than the preset threshold 1e-4, or the maximum number of iterations 300 has been reached.

[0075] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0076] 1. Large optical flow difference values ​​in image frames often indicate large motion amplitudes in the image, while small optical flow difference values ​​often indicate small motion amplitudes in the image. For video clips with large motion amplitudes, only a few frames need to be extracted as key frames to show the general process of the motion. However, for video clips with subtle motions in assembly line production, such as finger operations, more frames need to be extracted as key frames to show a more complete operation process. Image frames are classified by optical flow difference values. Fewer key frames are extracted for classes with large optical flow difference values, and more key frames are extracted for classes with small optical flow difference values. In this way, the extracted key frames can more completely show the operation process of assembly line production while ensuring the compression rate.

[0077] If the absolute difference between the optical flow difference value of one frame and the subsequent frame is large, it indicates that the frame likely records the transition from one action to another. Considering such frames as keyframes, rather than relying solely on clustering to extract keyframes, can enhance the coherence of the actions between keyframes.

[0078] 3. Group the image frames of the same category according to the difference in numbers so that the key frames in the same group have similar numbers. Then extract the key frames from each group of image frames separately. This enhances the continuity of the actions between the key frames and makes the distribution of the key frames in the original video more even, preventing the extracted key frames from being too concentrated in a certain video clip. At the same time, grouping for clustering also reduces the time complexity of the algorithm, reduces the time required for clustering, and improves processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Flow chart of the method of the present invention.

[0080] Figure 2 This is the k-means clustering flow chart in the present invention.

[0081] Figure 3 This is a clustering effect diagram in step S4 of the present invention.

[0082] Figure 4 This is a distribution diagram of key frame numbers in the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0084] Example 1:

[0085] refer to Figures 1 to 4 As shown, the present invention discloses a pipeline action video key frame extraction method based on optical flow estimation and ORB, comprising the following steps:

[0086] S1, input pipeline action video and grayscale the video;

[0087] S2. Calculate the optical flow difference value. Starting from the second frame, use the DIS algorithm to generate an optical flow vector for each pixel between the previous frame and the current frame. Then calculate the optical flow difference value between the current frame and the previous frame. The optical flow difference value of the first frame is the same as that of the second frame by default. In this way, each frame corresponds to an optical flow difference value. At this time, each image frame has two attributes: number and optical flow difference value.

[0088] S3. Find a key frame based on a significant change in the optical flow difference value. Traverse each image frame except the first and last frames. If the absolute value of the difference between the optical flow difference value of the current image frame and the optical flow difference value of the next frame is greater than or equal to r, then the current image frame is considered a key frame. The first and last frames are considered key frames by default.

[0089] S4. Clustering based on optical flow difference values: Perform k-means clustering based on the optical flow difference values ​​of each frame, clustering into x classes, calculating the proportion of the class center value of each class to the sum of all class center values, and using a dictionary to store the class number and the proportion of its class center value. At this point, each image frame has three attributes: number, category, and optical flow difference value;

[0090] S5. Extract local features of each image frame using the ORB algorithm;

[0091] S6. Convert each image frame into a feature vector based on the local features of the image frame extracted by the ORB algorithm. At this point, each image frame has four attributes: number, category, feature vector, and optical flow difference value;

[0092] S7. Group image frames of the same category according to their numbers: Arrange image frames of the same category in ascending order of numbers, traverse each frame, and the first frame of each category is defaulted to the group with group number 0 in the category. Subtract the number value of the previous frame from the number value of the current frame. If the difference is less than or equal to m, the current frame and the previous frame are classified into the same group. Otherwise, the group number is increased by one, and the current frame is classified into another group. At this point, each image frame has five attributes: number, category, group, feature vector, and optical flow difference value;

[0093] S8. Calculate the information entropy of each image frame. At this point, each image frame has six attributes: number, category, group, feature vector, information entropy, and optical flow difference value.

[0094] S9. Clustering based on eigenvectors: Perform k-means clustering on each group based on the eigenvectors. The number of clusters for each group is rounded up, and the frame with the largest information entropy is selected as the key frame. The number of clusters for each group is: the current group length / (a ​​+ the proportion of the center value of the class to which the current group belongs * b);

[0095] S10: Merge the key frame sets obtained in steps S3 and S9, and compose a new video from all key frames in ascending order of number.

[0096] In the present invention, a large optical flow difference value of an image frame often indicates a large amplitude of movement in the image, and a small optical flow difference value often indicates a small amplitude of movement in the image. For video clips with large amplitude of movement, only a few frames need to be extracted as key frames to show the general process of the movement. For video clips with subtle movements in assembly line production, such as finger operations, more frames need to be extracted as key frames to show a more complete operation process. The image frames are classified and processed by the optical flow difference value, and fewer key frames are extracted for the class with large optical flow difference value, and more key frames are extracted for the class with small optical flow difference value. The key frames extracted in this way can more completely show the operation process of assembly line production while ensuring the compression rate.

[0097] Example 2;

[0098] In step S1, the image is converted from the BGR color space to a grayscale image using the cv2.cvtColor function in the OpenCV library.

[0099] In step S2, a dictionary is used to store the attributes of the frame, with the frame number as the key and the dictionary {"flow_diff": optical flow difference value} as the value. The specific process of calculating the optical flow difference value is:

[0100] Step 1: Use the DIS algorithm to generate an optical flow vector for each pixel between the previous frame and the current frame. The optical flow vector describes the movement of these points between the two frames. The optical flow vector can be expressed as ( u , v ), u is the pixel displacement in the horizontal direction, v is the pixel displacement in the vertical direction, and the optical flow vector is calculated using the cv2.DISOpticalFlow_create function in the OpenCV library;

[0101] The DIS algorithm process includes a given template patch in the reference image, and DIS finds the best matching subwindow in the query image by gradient descent. The goal of this process is to find a transformation vector u , so that the sum of the squared differences between the template and query positions in the subwindow is minimized, the expression is:

[0102] ,in T is the template patch in the reference image, I t+1 is the query image, x is the pixel position in the patch, u is the best matching transformation vector obtained;

[0103] DIS uses a multi-scale approach to gradually optimize patches from coarse to fine. At each scale, the displacement of the patch is initialized by the displacement of the previous scale, and then optimized. The displacement of each scale is combined into a dense displacement field through weighted averaging. The mathematical expression is:

[0104] ,in, U s ( x ) is each pixel x The dense displacement field ，N s is the number of patches at that scale, d i ( x) is the patch and the deformed image in pixels x The intensity difference at u i It's a patch i The estimated displacement, λ ix is an indicator function that indicates that the patch i Whether with location x overlapping;

[0105] DIS uses variational refinement to further improve the accuracy of the displacement field. This step is achieved by minimizing an energy function that includes a data term and a smoothing term. The mathematical expression of the energy function is:

[0106] ,in, E ( U ) is the energy to be minimized, Ψ is a robust penalty function, E I 、 E G 、 E S They are intensity data term, gradient data term and smoothing term;

[0107] Through the above steps, a dense optical flow field describing the displacement of each pixel from one frame to another is finally generated, and the optical flow vector of each pixel between the previous frame and the current frame is obtained from the generated dense optical flow field ( u , v );

[0108] Step 2: Calculate the modulus of each optical flow vector, expressed as:

[0109] ;

[0110] Step 3: Calculate the average value of the optical flow vector modulus, that is, add up the modulus of the optical flow vectors of all pixels and divide it by the total number of pixels. This value is used as the optical flow difference value of the current frame, expressed as:

[0111] , use a dictionary to store the attributes of the frame. The elements of the dictionary are also dictionaries, with the number as the key and the dictionary {"flow_diff": optical flow difference value} as the value.

[0112] In step S4, if the center value of a certain category is much larger than the center values ​​of other categories, its center value is replaced by the second largest center value to prevent the center value of this category from being too high and the center value proportions of other categories from being too low.

[0113] In step S5, the specific process of using the ORB algorithm to extract local features is as follows:

[0114] Step 1: Construct a scale space pyramid. Specifically, set a scale factor scaleFactor and the number of pyramid levels nlevels, and reduce each frame image into nlevels images according to the scale factor.

[0115] Step 2, FAST key point detection, specifically for each pixel point p in the nlevels image of the scale space pyramid of each frame image, with pixel p as the center, select 16 pixels on a circle with a radius of 3, if there are N consecutive points with brightness greater than I on the 16 pixels on the circle, p +T or less than I p -T, then pixel p can be considered as a feature point (N=9), and all feature points are used as FAST feature points of the current frame;

[0116] Step 3: Orient the key points by using the intensity centroid. Specifically, the ORB algorithm uses the vector starting from the key point and ending at the intensity centroid as the direction of the key point. For each key point, its direction θ is calculated as follows:

[0117] ;

[0118] This formula defines a moment, through which the intensity centroid can be found, where I(x,y) is the pixel intensity value at coordinate (x,y), p and q are the order of the moment, and the centroid C Obtained by the following formula:

[0119] ;

[0120] in, m 00 Represents the sum of all pixel intensity values ​​in the image; m 10 is the sum of the products of the x-coordinates of all pixels in the image and their intensity values, m 01 It is the sum of the products of the y-coordinates of all pixels in the image and their intensity values;

[0121] The orientation θ of a keypoint is determined by the angle of the vector from the keypoint to its centroid:

[0122] ;

[0123] Step 4: Generate a rotation-invariant BRIEF descriptor. Specifically, the image is smoothed by Gaussian filtering, and then an S×S neighborhood (S=31) is taken around the key point. In the neighborhood, n point pairs (p,q) are selected by a selected method, and the point pairs are numbered starting from 1 in the order of their generation. The number is set to i, and the coordinate system of the key point is rotated according to the direction. θThe coordinates of the pixel pair (p′, q′) after rotation are calculated using the following rotation matrix:

[0124] ;

[0125] The pixel pair (p′, q′) inherits the number value i of the corresponding point pair (p, q). For each adjusted pixel pair (p′, q′), their intensity values ​​are compared. If I(p′)>I(q′), the i-th bit of the descriptor is set to 1, otherwise, the i-th bit of the descriptor is set to 0, and finally an n-bit binary string is generated.

[0126] Here, a scale factor scaleFactor (usually 1.2) and the number of pyramid levels nlevels (usually 8) are set. Each frame image is reduced to nlevels images according to the scale factor.

[0127] Meanwhile, step S5 can be implemented with the help of cv2.ORB_create() function in the OpenCV library.

[0128] Example 3;

[0129] In step S6, the specific process of using the BOF algorithm to convert each frame image into a feature vector is as follows:

[0130] Step 1: Construct visual vocabulary, specifically set F={f1, f2,…, f N} is the set of all extracted ORB descriptors, feature clustering: use the k-means clustering algorithm to cluster F into K clusters. The k-means goal is to minimize the sum of the squares of the distances from each sample point to its cluster center, expressed as:

[0131] ;

[0132] The center c of each class j Represents a visual word, the set of cluster centers {c1,c2,…,c K} constitutes a vocabulary dictionary; Step 2, feature vectorization, specifically the ORB feature descriptor set F for each image frame I Each descriptor f in i , calculate its distance to all visual words, and then classify it into the nearest visual word, expressed as:

[0133] ;

[0134] A K-dimensional feature vector h is constructed for each image frame, and the j-th dimension represents the ORB feature descriptor of the image frame classified into the visual vocabulary c j The frequency of the ORB feature descriptor f of the image framei Assigned to the visual word c j , then the value of the eigenvector in the jth dimension increases by 1, let h j Represents the value of the j-th dimension of the histogram, expressed as:

[0135] ;

[0136] Among them, l() is the indicator function;

[0137] Step 3: Normalize the feature vector. Specifically, use the L2 norm to make the modulus of the feature vector 1, which is expressed as:

[0138] ;

[0139] .

[0140] In step S8, the specific process of calculating information entropy is as follows:

[0141] Step 1: Histogram calculation: Assume that the image frame has N pixels and the grayscale range is 0 to L−1. Then the histogram H(i) of the image represents the number of pixels with grayscale level i in the image.

[0142] Step 2: Calculate the probability distribution. Specifically, calculate the probability distribution based on the histogram, that is, the probability P(i) of each gray level appearing. Divide each value in the histogram by the total number of pixels N, and express it as:

[0143] ;

[0144] Among them, P(i) represents the gray level in the image i The probability of the pixel appearing;

[0145] Step 3: Calculate information entropy, expressed as:

[0146] .

[0147] In step S9, parameter a is used to ensure the basic compression rate, and parameter b determines the difference in compression rate of image frames of different categories. The larger b is, the greater the difference in compression rate between categories with large optical flow difference values ​​and categories with small optical flow difference values, and vice versa.

[0148] If the absolute difference between the optical flow difference value of one frame and the optical flow difference value of the following frame is large, it indicates that the frame is likely recording the transition from one action to another. Considering such frames as keyframes, rather than relying solely on clustering to extract keyframes, can enhance the coherence of the actions between keyframes.

[0149] The k-means clustering in steps S4, S6 and S9 can be performed using the Kmeans function of the scikit-learn library. The specific process is as follows: Step 1, determine the number of clusters and initialize the cluster centers, which can be specifically divided into: Step A, set the number of clusters to k, and randomly select a sample as the first cluster center; Step B, calculate each sample point x The Euclidean distance to the nearest cluster center is denoted as D ( x ), in n-dimensional space, two points P 1( x 11 , x 12 ,…, x 1n )and P 2( x 21 , x 22 ,…, x 2n ) is calculated as follows: ; Step C, based on the distance calculated in step B, for each sample point that is not selected as the cluster center x Calculate a selection probability, expressed as: ;

[0150] Step D, repeat steps B and C until k cluster centers are selected;

[0151] Step 2: Assign the sample to the nearest cluster center: Calculate the Euclidean distance from each sample to each cluster center, and then assign the sample to the cluster represented by the nearest cluster center;

[0152] Step 3: Update the cluster center: After all samples are assigned to clusters, update the cluster center of each cluster. The new cluster center is the average value of the coordinates of the sample points it contains.

[0153] Step 4: Repeat steps 2 and 3 until the convergence condition is met: that is, the change in the cluster center is less than the preset threshold 1e-4, or the maximum number of iterations 300 is reached.

[0154] Image frames of the same category are grouped according to the difference in numbers, so that the key frames in the same group are numbered similarly. Then, key frames are extracted from each group of image frames separately. This enhances the continuity of actions between key frames and makes the distribution of key frames in the original video more even, preventing the extracted key frames from being too concentrated in a certain video segment. At the same time, grouping for clustering also reduces the time complexity of the algorithm, reduces the time required for clustering, and improves processing efficiency.

[0155] Although the present invention has been described herein with reference to a number of illustrative embodiments thereof, it will be understood that numerous other modifications and implementations may be devised by those skilled in the art that fall within the scope and spirit of the principles disclosed herein. More specifically, within the scope of the present disclosure, the drawings, and the claims, numerous variations and modifications may be made to the components and / or layout of the subject combination arrangement. In addition to variations and modifications to the components and / or layout, other uses will also be apparent to those skilled in the art.

Claims

1. A pipeline action video key frame extraction method based on optical flow estimation and ORB, characterized by: The following steps are involved: S1, input pipeline action video and grayscale the video; S2. Calculate the optical flow difference value. Starting from the second frame, use the DIS algorithm to generate an optical flow vector for each pixel between the previous frame and the current frame. Then calculate the optical flow difference value between the current frame and the previous frame. The optical flow difference value of the first frame is the same as that of the second frame by default. In this way, each frame corresponds to an optical flow difference value. At this time, each image frame has two attributes: number and optical flow difference value. S3. Find a key frame based on a significant change in the optical flow difference value. Traverse each image frame except the first and last frames. If the absolute value of the difference between the optical flow difference value of the current image frame and the optical flow difference value of the next frame is greater than or equal to r, then the current image frame is considered a key frame. The first and last frames are considered key frames by default. S4. Clustering based on optical flow difference values: Perform k-means clustering based on the optical flow difference values ​​of each frame, clustering into x classes, calculating the proportion of the class center value of each class to the sum of all class center values, and using a dictionary to store the class number and the proportion of its class center value. At this point, each image frame has three attributes: number, category, and optical flow difference value; S5. Extract local features of each image frame using the ORB algorithm; S6. Convert each image frame into a feature vector based on the local features of the image frame extracted by the ORB algorithm. At this point, each image frame has four attributes: number, category, feature vector, and optical flow difference value; S7. Group image frames of the same category according to their numbers: Arrange image frames of the same category in ascending order of numbers, traverse each frame, and the first frame of each category is defaulted to the group with group number 0 in the category. Subtract the number value of the previous frame from the number value of the current frame. If the difference is less than or equal to m, the current frame and the previous frame are classified into the same group. Otherwise, the group number is increased by one, and the current frame is classified into another group. At this point, each image frame has five attributes: number, category, group, feature vector, and optical flow difference value; S8. Calculate the information entropy of each image frame. At this point, each image frame has six attributes: number, category, group, feature vector, information entropy, and optical flow difference value. S9. Clustering based on eigenvectors: Perform k-means clustering on each group based on the eigenvectors. The number of clusters for each group is rounded up, and the frame with the largest information entropy is selected as the key frame. The number of clusters for each group is: the current group length / (a ​​+ the proportion of the center value of the class to which the current group belongs * b); S10, merging the key frame sets obtained in steps S3 and S9, and composing a new video by numbering all the key frames in ascending order; In step S9, parameter a is used to ensure the compression rate, and parameter b determines the difference in compression rates of image frames of different categories. The larger b is, the greater the difference in compression rate between categories with large optical flow difference values ​​and categories with small optical flow difference values, and vice versa.

2. The pipeline action video key frame extraction method based on optical flow estimation and ORB according to claim 1 is characterized in that: In step S1, the image is converted from the BGR color space to a grayscale image using the cv2.cvtColor function in the OpenCV library.

3. The pipeline action video key frame extraction method based on optical flow estimation and ORB according to claim 1 is characterized in that: In step S2, a dictionary is used to store the attributes of the frame, with the number as the key and the dictionary {"flow_diff": optical flow difference value} as the value. The specific process of calculating the optical flow difference value is as follows: Step 1, use the DIS algorithm to generate an optical flow vector for each pixel between the previous frame and the current frame. The optical flow vector describes the movement of these points between the two frames. The optical flow vector can be expressed as ( u , v ), u is the pixel displacement in the horizontal direction, v It is the pixel displacement in the vertical direction, and the optical flow vector is calculated using the cv2.DISOpticalFlow_create function in the OpenCV library; Among them, the DIS algorithm process includes a template patch in a given reference image, and DIS finds the best matching subwindow in the query image by gradient descent. The goal of this process is to find a transformation vector u , so that the sum of the squared differences between the template and query positions in the subwindow is minimized, the expression is: ,in T is the template patch in the reference image, I t+1 is the query image, x is the pixel position in the patch, u is the best matching transformation vector obtained; DIS uses a multi-scale method to gradually optimize the patch from coarse to fine. At each scale, the displacement of the patch is initialized by the displacement of the previous scale, and then optimized. The displacement of each scale is combined into a dense displacement field through weighted averaging. The mathematical expression is: ,in, U s ( x ) is each pixel x The dense displacement field ，N s is the number of patches at that scale, d i ( x ) is the patch and the deformed image in pixels x The intensity difference at u i It's a patch i The estimated displacement, λ ix is an indicator function that indicates that the patch i Whether with location x Overlap; DIS uses variational refinement to further improve the accuracy of the displacement field. This step is achieved by minimizing the energy function containing data terms and smoothing terms. The mathematical expression of the energy function is: ,in, E ( U ) is the energy to be minimized, Ψ is a robust penalty function, E I 、 E G 、 E S They are intensity data item, gradient data item and smoothing item respectively; through the above steps, a dense optical flow field describing the displacement of each pixel from one frame to another is finally generated, and the optical flow vector of each pixel point between the previous frame and the current frame is obtained from the generated dense optical flow field ( u , v ); Step 2, calculate the modulus of each optical flow vector, expressed as: Step 3: Calculate the average value of the optical flow vector modulus, that is, add up the modulus of the optical flow vectors of all pixels and divide it by the total number of pixels. This value is used as the optical flow difference value of the current frame, expressed as: , use a dictionary to store the attributes of the frame. The elements of the dictionary are also dictionaries, with the number as the key and the dictionary {"flow_diff": optical flow difference value} as the value.

4. The pipeline action video key frame extraction method based on optical flow estimation and ORB according to claim 1 is characterized in that: In step S4, if the center value of a certain category is much larger than the center values ​​of other categories, its center value is replaced by the second largest center value to prevent the center value of this category from being too high and the center value proportions of other categories from being too low.

5. The method for extracting key frames from pipelined action videos based on optical flow estimation and ORB according to claim 1, wherein: In step S5, the specific process of using the ORB algorithm to extract local features is as follows: Step 1, constructing a scale space pyramid, specifically setting a scale factor scaleFactor and the number of pyramid layers nlevels, and reducing each frame image into nlevels images according to the scale factor; Step 2, FAST key point detection, specifically for each pixel point p in the nlevels images of the scale space pyramid of each frame image, with pixel p as the center, select 16 pixels on a circle with a radius of 3, if there are N consecutive points with brightness greater than I on the 16 pixels on the circle, the key point is detected. p +T or less than I p -T, then pixel p can be considered as a feature point (N=9), and all feature points are used as FAST feature points of the current frame; Step 3, orient the key point by intensity centroid. Specifically, the ORB algorithm uses a vector with the key point as the starting point and the intensity centroid as the end point as the direction of the key point. For each key point, its direction θ is calculated as follows: ; This formula defines a moment, through which the intensity centroid can be found, where I(x,y) is the pixel intensity value at coordinate (x,y), p and q are the order of the moment, and the centroid C Obtained by the following formula: ; in, m 00 Represents the sum of all pixel intensity values ​​in the image; m 10 is the sum of the products of the x-coordinates of all pixels in the image and their intensity values, m 01 is the sum of the y-coordinates of all pixels in the image multiplied by their intensity values; the direction θ of a keypoint is determined by the angle of the vector from the keypoint to its centroid: ; Step 4, generate a rotation-invariant BRIEF descriptor, specifically by first smoothing the image through Gaussian filtering, then taking an S×S neighborhood (S=31) around the key point, and selecting n point pairs (p,q) in the neighborhood by a selected method, and numbering the point pairs starting from 1 in the order of point pair generation, and setting the number to i, and transforming the coordinate system of the key point according to the direction θ The coordinates of the pixel pair (p′, q′) after rotation are calculated using the following rotation matrix: ; The pixel pair (p′,q′) inherits the number value i of the corresponding point pair (p,q). For each adjusted pixel pair (p′,q′), compare their intensity values. If I(p′)>I(q′), the i-th bit of the descriptor is set to 1, otherwise, the i-th bit of the descriptor is set to 0, and finally generate an n-bit binary string.

6. The pipeline action video key frame extraction method based on optical flow estimation and ORB according to claim 1, characterized in that: In step S6, the specific process of using the BOF algorithm to convert each frame image into a feature vector is as follows: Step 1, construct visual vocabulary, specifically, set F = {f1, f2,…, f N } is the set of all extracted ORB descriptors, feature clustering: use the k-means clustering algorithm to cluster F into K clusters. The k-means goal is to minimize the sum of the squares of the distances from each sample point to its cluster center, expressed as: ; The center c of each class j Represents a visual word, the set of cluster centers {c1,c2,…,c K } constitutes a vocabulary dictionary; Step 2, feature vectorization, specifically the ORB feature descriptor set F for each image frame I Each descriptor f in i , calculate its distance to all visual words, and then classify it into the nearest visual word, expressed as: ; Construct a K-dimensional feature vector h for each image frame, where the j-th dimension represents the ORB feature descriptor of the image frame classified into the visual vocabulary c j The frequency of the ORB feature descriptor f of the image frame i Assigned to the visual word c j , then the value of the eigenvector in the jth dimension increases by 1, let h j Represents the value of the j-th dimension of the histogram, expressed as: ; Wherein, l() is an indicator function; Step 3, eigenvector normalization, specifically using the L2 norm to make the modulus of the eigenvector 1, expressed as: ; .

7. The pipeline action video key frame extraction method based on optical flow estimation and ORB according to claim 1, characterized in that: In step S8, the specific process of calculating information entropy is as follows: Step 1, histogram calculation, specifically: assume that the image frame has N pixels and the grayscale range is 0 to L-1, then the histogram H(i) of the image represents the number of pixels with grayscale level i in the image; Step 2, probability distribution calculation, specifically, calculate the probability distribution based on the histogram, that is, the probability P(i) of each grayscale level appearing, and divide each value in the histogram by the total number of pixels N, expressed as: ; Where P(i) represents the gray level in the image i The probability of the pixel appearing; Step 3, information entropy calculation, expressed as: .

8. The pipeline action video key frame extraction method based on optical flow estimation and ORB according to claim 1, characterized in that: The k-means clustering in steps S4, S6, and S9 uses the Kmeans function of the scikit-learn library for clustering. The specific process is as follows: Step 1, determine the number of clusters and initialize the cluster centers, which can be specifically divided into: Step A, set the number of clusters to k, and randomly select a sample as the first cluster center; Step B, calculate each sample point x The Euclidean distance to the nearest cluster center is denoted as D ( x ), in n-dimensional space, two points P 1( x 11 , x 12 ,…, x 1n )and P 2( x 21 , x 22 ,…, x 2n ) is calculated as follows: ; Step C: Based on the distance calculated in step B, for each sample point that is not selected as the cluster center x Calculate a selection probability, expressed as: ; Step D, repeat steps B and C until k cluster centers are selected; Step 2, assign samples to the nearest cluster center: calculate the Euclidean distance of each sample to each cluster center, and then assign the sample to the cluster represented by the nearest cluster center; Step 3, update the cluster center: after all samples are assigned to clusters, update the cluster center of each cluster, and the new cluster center is the average of the coordinates of the sample points it contains; Step 4, repeat steps 2 and 3 until the convergence condition is met: that is, the change of the cluster center is less than the preset threshold 1e-4, or the maximum number of iterations 300 has been reached.

Images