A method for extracting video high-definition key frames of effective targets of a substation panel

Abstract

The application discloses a kind of video high definition key frame extraction methods of effective target of substation panel, and extraction method steps include step one is based on Shi-Tomasi rectangular corner point algorithm Angle detection, identify the feature corner point at the junction of target panel frame profile and filter out the corner point capable of determining position.Step two is tracked to the feature corner point by straight line test using improved L-K optical flow method, with the mutation of feature corner point number as criterion, based on equal time interval collection method, extract the complete panel slow-moving first clear frame.Step three is based on feature corner point and carry out hierarchical nested cutting on the effective area on first clear key frame, hierarchical nested cutting effective rectangular area, carry out second clear frame screening to the same effective area.The key frame of the present application is extracted effectively, which has less redundancy and summarizes video content, and solves the shortcomings of data redundancy in traditional process monitoring.

Description

TECHNICAL FIELD

[0001] The application relates to a key frame extraction method, in particular to a video high-definition key frame extraction method for effective targets of a transformer substation panel. BACKGROUND

[0002] Video recognition refers to a new cross science of emerging technologies by using information processing and computer technology, pattern recognition, mathematical morphology and various methods to realize video image processing, analysis and understanding. With the continuous optimization of new algorithms of image recognition and video recognition derived therefrom and the improvement of computer computing capacity, the application of image and video recognition is becoming more and more extensive.

[0003] In the past years, with the intelligentization of power system operation, video technology has been gradually applied to the power field. Common video types can be divided into structured video and non-structured video. The former can find the segmentation point through obvious shot boundaries, and the latter needs a more intelligent method to judge the video content to obtain the boundary. Video is composed of continuous frames, and the key frame represents the main content described in the video. The extraction of the key frame is to extract the frame image that can best represent the video content from the video by using an effective algorithm, which has great significance in practical application scenarios. The extraction of the key frame is usually divided into two steps, that is, first, the video is segmented into several shots, and then the key frame is selected for the segmented shot.

[0004] In the field of power system fault diagnosis, the key frame, as a computer animation term, refers to the frame of a key action in the movement and change of a character or object. Now it is also introduced into the branch of video recognition technology. The information redundancy between similar frames is high. The application provides a video high-definition key frame extraction method for effective targets of a transformer substation panel. SUMMARY

[0005] The application aims to provide a video high-definition key frame extraction method for effective targets of a transformer substation panel. The Shi-Tomasi algorithm is used for corner detection to identify the intersection corner points of the panel frame contour, which have obvious changes in the gray gradient. A feature point mutation criterion is proposed. The improved and simplified L-K optical flow method is used to track the feature points. The mutation of the number of feature points is used as a criterion to quickly locate the complete panel slow-moving primary clear frame. The effective area is cut out by layering and nesting. The secondary clear frame screening is performed on a single effective area to complete the video high-definition key frame extraction of the effective target and solve the defect of data redundancy in the traditional process monitoring.

[0006] The purpose of the application can be achieved by the following technical scheme.

[0007] A video high-definition key frame extraction method for effective targets of a transformer substation panel, the extraction method comprising the following steps:

[0008] Step one: based on Shi-Tomasi rectangular corner point algorithm for corner detection, identify the feature corner of the target panel frame contour intersection and filter out the corner points that can determine the position.

[0009] Step two: using the improved L-K optical flow method to track the feature corner points through the straight line test, taking the mutation of the number of feature corner points as the criterion, based on the equal time interval collection method to extract the complete panel slow-moving primary clear frame.

[0010] Step three: based on the feature corner point, the effective area on the primary clear key frame is layered and nested cut, the effective rectangular area is cut, and the same effective area is twice clear frame screening.

[0011] Further, in step one, based on the Shi-Tomasi rectangular corner point algorithm for corner detection of video, the pixel gray value of (x, y) point is recorded as I(x, y), and the fixed size sliding window with (x, y) as the center is W(x, y). When the window moves a distance of (u, v), the pixel gray value change in the window can be expressed as:

[0012]

[0013] Because the effective target in the video has a large color difference with the invalid area, the feature corner points that distinguish the two can be identified based on the gray value change. In order to simplify the operation, formula 1 uses Taylor expansion Approximately expressed as:

[0014]

[0015] The matrix M in the formula is expressed as:

[0016]

[0017] Where the change of gray value depends on the matrix M, and its eigenvalues are λ1, λ2. Introducing the corner response value R = min(λ1, λ2) and the threshold value r, taking R>r as the criterion for the existence of corner points. The corner point extraction in the panel video is easily affected by the complex background environment of the panel, resulting in a large number of corner points, greatly increasing the calculation amount of the later processing. Therefore, an effective corner point suppression mechanism is introduced. If the number of corner points in the window is greater than the set corner point number threshold, the minimum interval of the corner point algorithm is adjusted to twice, until the maximum number of corner points in the region is not greater than the corner point number threshold.

[0018] Further, the tracking and collection process in step two is as follows:

[0019] S1: meaningless corner point elimination and feature corner point selection

[0020] The gradient value around the corner point changes sharply, and there are corner points around the endpoints of most line segments and line segment connections. Some meaningless line segments have less obvious features, smaller gradient change rates, and fewer surrounding corner points. Therefore, the extraction method of the application uses the LSD line extraction algorithm, introduces a corner point suppression mechanism to make the maximum number of corner points in the region not greater than the corner point number threshold, and then uses a connected domain area threshold to filter out a large number of invalid corner points. For the remaining corner points, according to the characteristics of the corner point distribution around the rectangle, some meaningless corner points that cannot form line segments are eliminated, and the meaningful corner points with more obvious features are retained. At the same time, the corner points near the intersection points of two perpendicular lines are taken as feature corner points, which are the targets of L-K optical flow tracking.

[0021] S2: tracking feature corner points using improved L-K optical flow method

[0022] The pyramid layering method is introduced to reduce the image size and reduce the target motion speed to improve the L-K optical flow method. Based on the initial position of the feature point in the process of extracting the motion of the board in the video, the improved L-K optical flow method is used to estimate the target optical flow information in each frame of image, to realize stable tracking of the feature corner points. Through the coordinate information of the target feature points in each frame of image and the initial distance between the feature points, the speed of the feature points and the distance between the feature points are calculated and saved in real time.

[0023] S3: determining complete and clear key frames through feature corner point threshold criterion, and marking this type of frames as first-level clear key frames.

[0024] Further, the basic assumption conditions of the optical flow method are:

[0025] 1) The brightness of the same target in different frames of images is constant.

[0026] 2) The motion distance of the target object between adjacent frames is relatively small.

[0027] 3) Spatial consistency, i.e., similar motion states exist for local pixel points in the image.

[0028] Further, the step three of the effective area layering and nesting cutting of the panel uses a layering and nesting labeling method:

[0029] (1) A larger rectangular label box is used to label the effective area which has continuity and cannot be clearly divided in the first step.

[0030] (2) In the first step, the regions with obvious features and relative independence are re-labeled within the bounding box to form a second layer of nested labeling, increasing the number of bounding boxes and achieving the effect of data enhancement.

[0031] Further, the step three panel effective area cutting result classification preprocessing adopts the regression subnetwork of YOLOv5 algorithm, utilizes the original YOLOv5 model to perform full connection and the like on the fused feature map, preliminarily judges the category and frame region information to which the effective area cutting result belongs, classifies different regions according to a database, generates accurate position information and performs cropping.

[0032] Further, the step three key frame secondary extraction technology for the difference feature of the region clearness performs convolution on the Sobol gradient function and the Brenner gradient function of the general clearness algorithm function:

[0033]

[0034] In the formula, M and N are picture resolutions, f(x,y) is a gray value of a point (x,y) in an image, f x (x,y) is a gradient value of the point (x,y) in the x direction, and f y (x,y) is a gradient value of the point (x,y) in the y direction.

[0035] Therefore, a double gradient convolution function F Doubel is obtained, which achieves an ideal unimodal and unbiased curve.

[0036]

[0037] Based on the function, a quantitative clearness difference index is obtained by evaluating the clearness of the image, and the greater the value, the higher the clearness, and the image frame greater than a certain threshold is output for secondary extraction.

[0038] The beneficial effects of the present application are as follows:

[0039] 1. The extraction method based on video processing technology pre-processes the on-site collected video to be detected, and uses the effective region picture screened out after key frame extraction to remove invalid information, thereby effectively improving the proportion of data in total data.

[0040] 2. The extraction method obtains a key frame set of the video, deletes redundant parts, and greatly reduces the occupied storage space.

[0041] 3. The extraction method effectively extracts key frames with less redundancy and summarized video content through optical flow fixed-point tracking, and the extraction method applied to power failure inspection and the like can play an important auxiliary role, and provides a monitoring platform for power failure inspection. BRIEF DESCRIPTION OF DRAWINGS

[0042] The application will be further described below with reference to the drawings.

[0043] Figure 1 is a flow chart of the extraction method of the application;

[0044] Figure 2 is the extraction of basic video structure and key frame;

[0045] Figure 3 is the result of corner point suppression and

[0046] Figure 4 is the result of meaningless corner point elimination screening;

[0047] Figure 5 is a comparison chart of high-definition key frame and ordinary key frame. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0049] In combination with Figures 1-5 shown, a video high-definition key frame extraction method of effective target of a substation panel is shown in Figure 1 The extraction method comprises the following steps:

[0050] Step 1: Perform corner point detection based on Shi-Tomasi rectangular corner point algorithm, identify the corner points at the intersection of the target panel frame contour, and screen out the corner points capable of determining the position;

[0051] Perform corner point detection on the video based on Shi-Tomasi rectangular corner point algorithm. The pixel gray value of the (x, y) point is denoted as I(x, y), and the fixed-size sliding window with (x, y) as the center is denoted as W(x, y). When the window moves by a distance of (u, v), the pixel gray value change in the window can be expressed as:

[0052]

[0053] Since the effective target in the video generally has a large color difference from the invalid area, the gray value change can be used to identify the characteristic corner points distinguishing the two. In order to simplify the calculation, formula 1 uses Taylor expansion to approximate as:

[0054]

[0055] The matrix M in the formula is expressed as:

[0056]

[0057] It can be seen that the change size of the gray value depends on the matrix M, the eigenvalues of which are λ1 and λ2, the corner response value R = min(λ1, λ2) and the threshold value r are introduced, and R>r is taken as the criterion for the existence of the corner point. The corner point extraction in the panel video is easily affected by the complex background environment in which the panel is located, a large number of corner points are generated, and the calculation amount of the post-processing is greatly increased. For this purpose, an effective corner point suppression mechanism is introduced, if the number of corner points in the window is greater than the set corner point number threshold, the minimum interval of the corner point algorithm is adjusted to twice, until the maximum number of corner points in the region is not greater than the corner point number threshold, the purpose of suppression is achieved, and the corner point detection result is as shown in Figure 3 The method can effectively label and record the panel and the four corners of the panel screen and the keys in the form of corner points, so as to facilitate the feature corner point in the later stage.

[0058] Step two: the feature corner points are tracked by using the improved L-K optical flow method, the mutation of the number of feature corner points is taken as the criterion, and the complete panel slow-moving primary clear frame is extracted based on the equal time interval collection method, wherein the tracking collection process is as follows:

[0059] S1: meaningless corner point elimination and feature corner point selection

[0060] The gradient value around the corner point changes sharply, the end points of most line segments and the connection positions of the line segments will exist corner points, and some meaningless line segments have relatively small gradient change rate and small number of surrounding corner points due to the relatively fuzzy features. Therefore, the extraction method of the application utilizes the LSD line extraction algorithm, introduces the corner point suppression mechanism, so that the maximum number of corner points in the region is not greater than the corner point number threshold, and then a large number of invalid corner points are filtered by using the connected domain area threshold, a large number of invalid corner points are filtered by using the connected domain area threshold, for the remaining corner points, some meaningless corner points which cannot form line segments are eliminated according to the distribution characteristics of the corner points around the rectangle, and the meaningful corner points with more obvious features are reserved, the screening result is as shown in Figure 4 Figure 3 The number of corner points is less than

[0061] S2: the feature corner points are tracked by using the improved L-K optical flow method

[0062] ​The basic assumption conditions of the optical flow method are: the brightness of the same target in different frames of pictures is constant; the movement distance of the target object between adjacent frames is relatively small; spatial consistency, that is, the local pixel points of the image have similar motion states.

[0063] Since there is a large error in tracking when the target moves at a high speed, a pyramid layering method is introduced to reduce the image size and thus reduce the target movement speed to improve the L-K optical flow method. The optical flow vector extracted by the original method is a two-dimensional vector describing the instantaneous speed of the point. Experiments show that the panel hardly moves longitudinally, so only lateral movement is tracked. Based on the initial position of the feature points in the process of the panel movement in the video to be extracted, the improved L-K optical flow method is used to estimate the target optical flow information in each frame of image, so as to realize stable tracking of the feature corner points. The speed of the feature points and the distance between the feature points are calculated and saved in real time through the coordinate information of the target feature points in each frame of image and the initial distance between the feature points.

[0064] S3: Determine the complete and clear key frame through the threshold criterion of the feature corner points, and mark this type of frame as a first-level clear key frame

[0065] As shown in Figure 2 , a video can usually be divided into the smallest structure in units of frames, and related frames can form the same scene. The key frame containing all the effective information in each scene is extracted, that is, the information storage space can be greatly compressed, and the data effective ratio can be effectively improved. Through real-time statistics of the number of corner points in the process of video playing, the experimental results of the sudden change of the number of corner points when the effective area of the panel appears completely are obtained. Since the effective area is not connected, it is most appropriate to use the extreme points of the feature corner point number curve as the judgment condition for the appearance of the key frame. The optical flow is the instantaneous speed of the pixel movement of the spatial moving object on the observation imaging plane. Due to the angle of the camera, the speed of the uniformly moving object is not a constant value from entering the video to leaving the video. The smaller the acceleration, the more clearly the shooting can be approximated. It is approximated that the acceleration α of the feature corner points in a frame obeys the Gaussian distribution with an expectation μ and a standard deviation σ.

[0066] The clear frame condition is set as:

[0067] μ+mσ<α<μ+nσ

[0068] Wherein, m, n are empirical parameters obtained after the experiment.

[0069] Step three: based on the feature corner points, the effective area on the first-level clear key frame is hierarchically nested and cut, the effective rectangular area is cut out by hierarchical nested cutting, the same effective area is screened for the second time, and the specific operation process is as follows:

[0070] (1) Hierarchical nested cutting of the panel effective area

[0071] The effective area cutting preprocessing adopts a hierarchical nested labeling method, a larger rectangular labeling box is used to label the effective area with continuity and cannot be clearly divided in the first step; then, the second labeling is performed on the area with obvious features and relative independence in the labeling box of the first step to form the second layer of nested labeling, thereby increasing the number of labeling boxes and achieving the effect of data enhancement.

[0072] (2) Panel effective area cutting result classification

[0073] The effective area classification preprocessing adopts the regression subnetwork of the YOLOv5 algorithm, uses the original YOLOv5 model to perform full connection and the like on the fused feature map, preliminarily judges the category and the bounding box area information to which the effective area cutting result belongs, classifies different areas according to the database, generates accurate position information and performs cropping.

[0074] (3) Cutting classification result clarity screening

[0075] The key frame secondary extraction technology aiming at the differentiating features of the area clarity performs convolution on the Sobol gradient function and the Brenner gradient function of the general clarity algorithm function:

[0076]

[0077] In the formula, M and N are the picture resolution, f(x, y) is the gray value of the (x, y) point in the image, f x (x, y) is the gradient value of the (x, y) point in the x direction, and f y (x, y) is the gradient value of the (x, y) point in the y direction.

[0078] Therefore, a double gradient convolution function F Doubel :

[0079]

[0080] Based on the function, the image clarity is evaluated to obtain a quantitative clarity difference index, and the larger the value is, the higher the clarity is. The image frames greater than a certain threshold value are output for secondary extraction, and the extraction method can be applied to process the video captured by the power fault automatic inspection, extract the effective high-definition pictures in the inspection video, and help to find and repair faults in real time

[0081] In the description of the specification, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0082] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application.

Claims

1. A method for extracting video high definition key frame of effective target of substation panel, characterized in that, The extraction method comprises the following steps: Step 1: based on Shi-Tomasi rectangular corner point algorithm, corner point detection is carried out, feature corner points at the junction of the target panel frame contour are recognized, and corner points capable of determining positions are screened out; Step 2: the feature corner points are tracked by using the improved L-K optical flow method, and based on the equal time interval collection method, a complete panel slow-moving primary clear frame is extracted, taking the mutation of the number of feature corner points as a criterion; Step 3: based on the feature corner points, an effective area on the primary clear key frame is subjected to hierarchical nested cutting, the effective rectangular area is cut out, and secondary clear frame screening is carried out on the same effective area; The tracking and collecting process in step 2 is as follows: S1: meaningless corner point elimination and feature corner point selection The gradient value around the corner point changes sharply, most of the end points of the line segment and the connection part of the line segment will exist corner points, and some meaningless line segments have relatively small gradient change rate and fewer surrounding corner points; by using the LSD line extraction algorithm, the corner point suppression mechanism is introduced, so that the maximum number of corner points in the region is not greater than the corner point number threshold, then a large number of invalid corner points are filtered out by using the connected domain area threshold, for the remaining corner points, according to the characteristics of the corner point distribution around the rectangle, some meaningless corner points which cannot constitute a line segment are eliminated, and the meaningful corner points with more obvious features are reserved, and the corner points near the intersection points of two perpendicular lines are taken as feature corner points, which are taken as the tracking target of the L-K optical flow method; S2: the feature corner points are tracked by using the improved L-K optical flow method The image size is reduced by introducing the pyramid hierarchical method, the target motion speed is reduced, the L-K optical flow method is improved, the initial position of the feature point in the process of the panel motion in the video to be extracted is taken as the basis, the improved L-K optical flow method is used to estimate the target flow information in each frame of image, the stable tracking of the feature corner points is realized, the speed of the feature points and the distance between the feature points are calculated and saved in real time through the coordinate information of the target feature points in each frame of image and the initial distance between the feature points; S3: the complete clear key frame is determined by the feature corner point threshold criterion, and the frame is marked as a primary clear key frame; In step 3, the effective area of the panel is subjected to hierarchical nested cutting by using the effective area cutting pretreatment: (1) a larger rectangular marking box is used to mark the effective area which has continuity and cannot be clearly divided in the first step; (2) in the marking box in the first step, the area which has very obvious features and has relative independence is subjected to secondary marking, the second layer of nested marking is formed, the number of marking boxes is increased, and the effect of data enhancement is achieved.

2. The method of claim 1, wherein the method further comprises: The step one is based on Shi-Tomasi rectangle corner point algorithm to detect the corner point of the video, The pixel gray value of the point is recorded as The fixed size sliding window with as the center is When the window moves a distance of , the pixel gray value change in the window can be expressed as: (Formula 1) Since the effective target in the video generally has a large color difference with the invalid area, the feature corner points can be distinguished based on the gray change amount, in order to simplify the operation, formula 1 uses Taylor expansion Approximately expressed as: The expression of the matrix M is: wherein the magnitude of the change in the gray value depends on the matrix , whose eigenvalues are , , the corner response value and the threshold value are introduced as the criterion for the existence of the corner point, the corner point extraction in the panel video is easily affected by the complex background environment where the panel is located, a large number of corner points are generated, and the calculation amount of the later processing is greatly increased, therefore, an effective corner point suppression mechanism is introduced, if the number of the corner points in the window is greater than the set corner point number threshold value, the minimum interval of the corner point algorithm is adjusted to twice, until the maximum number of the corner points in the region is not greater than the corner point number threshold value.

3. The method of claim 1, wherein the method further comprises: The basic assumption conditions of the optical flow method are: 1) the brightness of the same target in different frames of images is constant; 2) the movement distance of the target object between adjacent frames is relatively small; 3) Spatial consistency, that is, the local pixels of the image have similar motion states.

4. The method of claim 1, wherein the method further comprises: The step three panel effective area cutting result classification preprocessing adopts a regression subnetwork of a YOLOv5 algorithm, the original YOLOv5 model is used for full connection processing of the fused feature map, the class to which the effective area cutting result belongs and the frame information are preliminarily judged, different regions are classified according to a database, accurate position information is generated, and cutting is performed.

5. The method of claim 4, wherein the method further comprises: The step three key frame secondary extraction technology for the difference of the region definition difference feature carries out convolution on a general definition algorithm function Sobol gradient function and a Brenner gradient function: where M and N are picture resolutions, is a gray value of a point in the image , is a gradient value of the point in the direction , x is a gradient value of the point in the direction , is a gradient value of the point in the direction y . Thus, a double gradient convolution function that achieves an ideal unimodal and unbiased curve : Based on the function, the image definition is evaluated to obtain a quantitative definition difference index, and the larger the value is, the higher the definition is. Through output of an image frame greater than a certain threshold, secondary extraction is carried out.

Images