A method and system for automatically extracting key frames from blast furnace material surface video

Abstract

The method and system for automatically extracting key frames of blast furnace material surface video provided by the present invention collects material surface video of blast furnace smelting process, locates the significant area of ​​material surface image based on boundary prior and image feature space, extracts accurate feature points of significant area of ​​image, and calculates pixel displacement of two adjacent frames of image according to optical flow method; then proposes a method of fusing local density maximum and GMM model clustering to identify material surface state; finally, divides the video collected by high-temperature industrial endoscope into different periods according to the change of density of image feature points, and extracts key frames of each material distribution period based on material surface state recognition result. The present invention can accurately identify material surface state and eliminate redundant information of material surface video, and automatically extract material surface video key frames with stable central airflow and obvious image features from blast furnace material surface video. At the same time, the present invention is suitable for automatic extraction of key frames of other videos with periodic changes and uneven image quality.

Description

Technical Field

[0001] The invention mainly relates to the technical field of blast furnace smelting, and in particular to a method and system for automatically extracting key frames from a blast furnace charge surface video. Background Art

[0002] The blast furnace is a large, closed black-box reaction vessel and a key equipment for producing molten iron. During the ironmaking process in a blast furnace, the charge (including iron ore, limestone, coke, etc.) is sprinkled into the furnace from the top of the furnace in a certain proportion. At the same time, the hot blast furnace blows hot air from the tuyere at the bottom of the blast furnace. Under the action of the buoyancy of the hot air and its own gravity, the charge will form a charge surface that changes within a certain height range on the top of the furnace. The coke in the charge burns at the tuyere to produce high-temperature reducing gas, and during the rising process, the iron ore is reduced to iron, forming molten iron in a high-temperature environment and discharged from the iron mouth. Obtaining clear blast furnace charge surface video is of great significance for real-time monitoring of the operating status of the furnace and avoiding abnormal operating conditions. However, considering that the blast furnace charge distribution is periodic and intermittent, the collected charge surface video also shows a periodic change pattern. Under normal operating conditions, the charge surface morphology changes relatively slowly, and the charge surface video of the same period has a lot of repeated and redundant information. In addition, due to the intermittent charge sprinkled from the top of the furnace and strong dust interference, the image quality of different periods in the video often varies greatly. Compared with the material surface images blocked by furnace charge, dust, etc., clear and stable material surface images can provide more accurate and reliable information for on-site personnel to timely grasp the operation status of the furnace, changes in material surface morphology and distribution of gas flow. However, in the process of collecting blast furnace material surface videos, on-site personnel need to spend a lot of time to obtain key information in the video, and if all frame images are analyzed and processed in the later stage, the efficiency will be greatly reduced. The key frame of blast furnace material surface video is a clear image sequence with stable central airflow and obvious features. The study of the automatic extraction method of blast furnace material surface video key frame can quickly screen out high-quality and representative video frames from a large number of material surface videos, greatly improving the accuracy and efficiency of later video processing.

[0003] The extraction of video key frames is an important means to reduce video redundancy and eliminate invalid and repeated information. It has wide applications in video retrieval, video compression storage, video summarization, video classification, industrial video monitoring and other fields. Existing video key frame extraction methods can be divided into methods based on image features, methods based on shot detection, methods based on clustering, and methods based on motion information. The image feature-based method calculates the difference of these features through the underlying features of the image such as color, texture, brightness, and motion, and compares them with the set threshold; the shot detection-based method first divides the video into many shots, and selects the first frame, the last frame, and several fixed frames in the middle as key frames; the clustering-based method first uses a clustering algorithm to divide similar frames into the same category, and selects representative key frames from different categories; the motion information-based method needs to detect images containing moving targets from the video. Considering the harsh furnace environment and the periodicity of the morphological changes of the blast furnace charge surface video, the non-uniformity of image quality, etc., compared with the extraction of key frames of ordinary scene videos, the automatic extraction of key frames of blast furnace charge surface videos is more complicated, and it has important industrial value for timely monitoring the changes in the operating status of the blast furnace smelting process.

[0004] Chinese patent CN114023345A "A method for extracting key frames from the tail of a sintering machine based on audio information" discloses a method for extracting key frames from the tail of a sintering machine using audio information. The audio signal of the sintering machine is denoised using Wiener filtering and spectral subtraction. The mutation position in the audio is extracted as the position of the dropped frame, and the next frame after the dropped frame is used as the dropped frame to extract the key frames from the tail of the sintering machine. This method mainly uses the audio information of sintered ore falling at the tail of the sintering machine, and is not suitable for extracting key frames from the blast furnace charge surface video.

[0005] Chinese patent CN107832694A "A video key frame extraction algorithm" discloses a video key frame extraction algorithm. The algorithm calculates the width, height and feature information of the effective image area of ​​a frame in the current input video stream, and compares the difference with the previous frame and the similarity of all frames in the cache area, obtains the difference frame through the set threshold, counts the variance value of the difference frame and outputs the key frame. This method mainly determines whether it is a key frame based on the image difference of the video frame, and needs to set the threshold in advance, which is not suitable for the extraction of key frames of blast furnace charge surface video with high dynamic changes. Summary of the invention

[0006] The method and system for automatically extracting key frames from a blast furnace material surface video provided by the present invention solve the technical problem that the prior art cannot automatically extract clear key frames with obvious image features from a blast furnace material surface video with periodic changes and uneven image quality.

[0007] In order to solve the above technical problems, the method for automatically extracting key frames from a blast furnace charge surface video proposed by the present invention comprises:

[0008] Obtain the salient area of ​​the material surface image;

[0009] Extract feature points of salient areas;

[0010] Match the feature points of two adjacent frames of material surface images, and calculate the number of feature points and the optical flow of feature points of the two adjacent frames of material surface images;

[0011] Based on kernel density estimation, the local density maximum of the number of feature points and the average optical flow vector data distribution is obtained to obtain the initial clustering center of the Gaussian mixture model, where the average optical flow vector is the average value of the optical flow of all feature points in the material surface image;

[0012] Based on the initial clustering center, the Gaussian mixture model is used to cluster the optical flow of the feature points of the material surface image to obtain the state of the material surface, where the state of the material surface includes fast sinking state, unstable state, stable state and slow sinking state;

[0013] The cloth cycle is divided according to the density change of the feature points of the material surface image, and the key frames in the videos of different cloth cycles are extracted based on the state of the material surface.

[0014] Furthermore, obtaining the salient areas of the material surface image includes:

[0015] The surface image is divided into superpixel blocks with a preset threshold through SLIC clustering;

[0016] Calculate the boundary connectivity between superpixel blocks and boundaries;

[0017] According to the boundary connectivity, the probability that the superpixel block belongs to the background is obtained;

[0018] According to the probability that the super pixel block belongs to the background, the saliency value of the super pixel block is obtained;

[0019] According to the saliency value of the superpixel block, the salient area of ​​the material surface image is obtained.

[0020] Furthermore, obtaining the salient area of ​​the material surface image according to the salient value of the superpixel block includes:

[0021] Based on the feature space of the material surface image, the overall difference between the pixels of the material surface image is obtained, and the calculation formula for obtaining the overall difference between the pixels of the material surface image is:

[0022]

[0023] Among them, d1(p i ,p j )、d2(pi ,p j ) and d3(p i ,p j ) represent the pixels p of the surface image respectively. i and pixel p j The Euclidean distance of color feature, brightness feature and texture feature, d s (p i ,p j ) represents pixel p i and pixel p j The Euclidean distance of the spatial features between them, α represents the parameter, which is set to 3;

[0024] According to the saliency value of the superpixel block and the overall difference between pixels, the saliency value of the pixel is obtained, wherein the calculation formula for obtaining the saliency value of the pixel is:

[0025]

[0026] Among them, S pi Represents pixel p i The significant value of d(p i ,p n ) represents pixel p i and pixel p n The overall difference, b pi Indicates that it contains pixel p i The superpixel block, N represents the pixel p i The number of pixels in the 8×8 neighborhood centered at pi ) represents the super pixel block b pi Probability of belonging to the background;

[0027] The material surface image is binarized according to the significant value of the pixel to obtain the significant area of ​​the material surface image.

[0028] Furthermore, extracting feature points of the salient area includes:

[0029] The regional images corresponding to the salient regions are downsampled to obtain regional images of different sizes, and the regional images of different sizes are arranged from large to small, and Gaussian difference pyramids are obtained through Gaussian convolution and Gaussian difference.

[0030] Compare each pixel of the image in the Gaussian difference pyramid with the adjacent pixels and the pixels in the adjacent areas of the upper and lower layers of the image to find the extreme points, thereby obtaining discrete extreme points;

[0031] The discrete extreme points are fitted into continuous extreme points through the Taylor formula, the unstable points are eliminated, and the edge influence is eliminated based on the Harris corner detection algorithm to obtain a continuous set of feature points, thereby obtaining the feature points of the significant area.

[0032] Furthermore, the calculation formula for calculating the number of feature points and the optical flow of feature points of two adjacent frames of material surface images is:

[0033] (u x ,v y )=(x+△xx,y+△yy)=(△x,△y),

[0034] Among them, u x and v y Respectively represent the i-th feature point P of the t-th frame surface image i (x, y) moves to the i-th feature point P′ of the t+1-th frame surface image i (x+△x,y+△y) is the motion component in the horizontal and vertical directions, x,y represents the feature point P i The coordinates of (x, y), x+△x, y+△y represent the feature point P′ i The coordinates of (x+△x,y+△y).

[0035] Furthermore, the number of feature points and the local density maximum of the average optical flow vector data distribution are obtained based on kernel density estimation, and the initial clustering center of the Gaussian mixture model is obtained, including:

[0036] Collect material surface images at different periods and in different states, and calculate the number of feature points and the average optical flow vector of two adjacent frames of images, where the average optical flow vector is the average value of the optical flow of all feature points in the material surface image;

[0037] Equal confidence probability is used to remove outliers, and the number of feature points and the average optical flow vector are normalized;

[0038] The kernel density estimation function is used to obtain the probability density map of the number of feature points and the average optical flow vector data distribution, where the kernel density estimation function is specifically:

[0039]

[0040] Among them, h represents the smoothing parameter, K h (·) represents the kernel function, d represents the observed value, and f h (d) represents the probability density function, M represents the total number of samples, and d i Represents the i-th sample in the feature space sample set;

[0041] According to the probability density map, the four points with the largest local density are obtained as the initial cluster centers, and they are used as the initial cluster centers of the Gaussian mixture model.

[0042] Furthermore, based on the initial cluster center, the Gaussian mixture model is used to perform state clustering on the feature point optical flow of the material surface image, and the state of the material surface is obtained, including:

[0043] The EM algorithm is used to update the parameters of the Gaussian mixture model until the log-likelihood function reaches its maximum value, and the parameter update ends. The log-likelihood function is specifically:

[0044]

[0045] Among them, α i represents the probability that the current frame belongs to category i, μ i ,∑ i Respectively represent the mean and covariance matrix of the i-th Gaussian distribution, f(α i ,μ i ,∑ i ) represents the objective function of parameter update, L(α i ,μ i ,∑ i ) represents the log-likelihood function, m represents the number of samples, k represents the number of Gaussian functions, x j represents the current sample, P(x j / μ i ) represents the sample x j The probability of belonging to the i-th Gaussian distribution;

[0046] Draw a Gaussian distribution curve and classify the data within or close to the Gaussian distribution curve into the same state.

[0047] Furthermore, the material cycle is divided according to the density change of the feature points of the material surface image, and the key frames in the videos of different material cycles are extracted based on the state of the material surface, including:

[0048] The density of feature points is obtained according to the average number of feature points of the material surface image per second;

[0049] Compare the density of feature points of the material surface image at the current moment with the density of feature points of the material surface image at the previous second, and divide the material distribution cycle according to a preset comparison threshold;

[0050] Calculate the Euclidean distance between the feature value of the current frame and different cluster centers to obtain the state of the current frame. If it is in a fast sinking state or an unstable state, it is determined to be a non-key frame and the frame number remains unchanged. If it is in a stable state or a slow sinking state, it is determined to be a candidate key frame and the frame number is increased by 1.

[0051] The sampling frequency is determined according to the number of candidate key frames to ensure that the number of key frames extracted in each cycle is the same. From the candidate key frames, the image frame corresponding to the cluster center, the image frame with the largest number of feature points and the smallest feature point optical flow, and the image obtained at a fixed sampling frequency are selected as key frames to obtain a set of key frames for a cycle.

[0052] The key frame sets of different material distribution cycles are extracted to obtain the blast furnace material surface video frame set.

[0053] The blast furnace material surface video key frame automatic extraction system provided by the present invention comprises:

[0054] A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for automatically extracting key frames from a blast furnace material surface video provided by the present invention are implemented.

[0055] Compared with the prior art, the advantages of the present invention are:

[0056] The method and system for automatically extracting key frames of blast furnace material surface video provided by the present invention collects material surface video of blast furnace smelting process by adopting high-temperature industrial endoscope, locates the significant area of ​​material surface image based on boundary prior and image feature space, extracts accurate feature points of significant area of ​​image, and calculates pixel displacement of two adjacent frames of image according to optical flow method; then proposes a method of fusing local density maximum and GMM model clustering to identify material surface state; finally, divides the video collected by high-temperature industrial endoscope into different periods according to the change of density of image feature points, and extracts key frames of each material distribution period based on material surface state recognition result. The present invention can accurately identify material surface state and eliminate redundant information of material surface video, and automatically extract material surface video key frames with stable central airflow and obvious image features from blast furnace material surface video. At the same time, the present invention is suitable for automatic extraction of key frames of other videos with periodic changes and uneven image quality.

[0057] The key points of the present invention include:

[0058] (1) Based on the boundary prior and image feature space, the significant values ​​of different pixels in the material surface image are obtained, and the significant area of ​​the material surface image is extracted, thereby greatly reducing the impact of non-material surface areas and gas flow changes on feature extraction;

[0059] (2) The density of feature points and the average pixel displacement are regarded as two key features for material surface image state recognition. The feature points of the salient areas of the material surface are extracted and refined matching is performed to obtain a set of feature points that correspond to each other in two adjacent frames, and the size of the optical flow vector of the feature points is calculated.

[0060] (3) The local density maximum and GMM model are integrated to cluster the number of feature points and the size of optical flow vectors of the material surface images at different periods, and the material surface images are divided into four states: unstable, stationary, slowly sinking, and quickly sinking;

[0061] (4) The material surface period is divided according to the density of feature points, and based on the results of state recognition, the key frames with stable airflow and obvious features in the center of the blast furnace material surface video are extracted;

[0062] (5) The present invention combines methods such as period division, motion information extraction, and state clustering, and can realize the automatic extraction of key frames of complex industrial videos with periodic changes, multiple states, and uneven image quality according to different needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The method for automatically extracting key frames from a blast furnace charge surface video according to the second embodiment of the present invention;

[0064] Figure 2 The result of extracting feature points of the salient area of ​​the material surface image according to the second embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of material distribution cycle division according to the second embodiment of the present invention;

[0066] Figure 4 Schematic diagram of the key frame extraction process of the surface video of the second embodiment of the present invention;

[0067] Figure 5 It is a structural block diagram of the automatic key frame extraction system of the blast furnace material surface video according to an embodiment of the present invention.

[0068] Reference numerals:

[0069] 10. Memory; 20. Processor. DETAILED DESCRIPTION

[0070] In order to facilitate the understanding of the present invention, the present invention will be described more comprehensively and meticulously below in conjunction with the accompanying drawings and preferred embodiments of the present invention, but the protection scope of the present invention is not limited to the following specific embodiments.

[0071] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0072] Embodiment 1

[0073] The method for automatically extracting key frames from a blast furnace charge surface video provided by the first embodiment of the present invention includes:

[0074] Step S101, obtaining a significant area of ​​a material surface image;

[0075] Step S102, extracting feature points of a salient area;

[0076] Step S103, matching feature points of two adjacent frames of material surface images, and calculating the number of feature points and the optical flow of feature points of the two adjacent frames of material surface images;

[0077] Step S104, obtaining the number of feature points and the local density maximum of the average optical flow vector data distribution based on kernel density estimation, and obtaining the initial cluster center of the Gaussian mixture model, wherein the average optical flow vector is the average value of the optical flow of all feature points in the material surface image;

[0078] Step S105, based on the initial cluster center, a Gaussian mixture model is used to perform state clustering on the feature point optical flow of the material surface image to obtain the state of the material surface, wherein the state of the material surface includes a fast sinking state, an unstable state, a stable state and a slow sinking state;

[0079] Step S106, dividing the material cycle according to the density change of the feature points of the material surface image, and extracting key frames in the videos of different material cycles based on the state of the material surface.

[0080] The method for automatically extracting key frames of blast furnace material surface video provided by the embodiment of the present invention obtains the significant area of ​​the material surface image, extracts the feature points of the significant area, matches the feature points of two adjacent material surface images, calculates the number of feature points and the optical flow of feature points of the two adjacent material surface images, obtains the number of feature points and the local density maximum of the average optical flow vector data distribution based on kernel density estimation, obtains the initial clustering center of the Gaussian mixture model, and based on the initial clustering center, uses the Gaussian mixture model to perform state clustering on the optical flow of the feature points of the material surface image, obtains the state of the material surface, and divides the material distribution cycle according to the change of the density of the feature points of the material surface image, and extracts the key frames in the video of different material distribution cycles based on the state of the material surface, solves the technical problem that the prior art cannot automatically extract clear key frames with obvious image features from the blast furnace material surface video with periodic changes and uneven image quality, can accurately identify the material surface state and eliminate the redundant information of the material surface video, and automatically extracts the material surface video key frames with stable central airflow and obvious image features from the blast furnace material surface video. At the same time, the present invention is applicable to the automatic extraction of other video key frames with periodic changes and uneven image quality.

[0081] Embodiment 2

[0082] Embodiment 2 of the present invention provides a method for automatically extracting clear key frames with obvious image features from a blast furnace charge surface video with periodic changes and uneven image quality. The overall idea is as follows: Figure 1 As shown, the following steps are included:

[0083] (1) Based on the boundary prior and image feature space, the salient areas of the material surface image are located to reduce the influence of non-material surface areas and gas flow changes in the material surface image.

[0084] (2) The density of feature points and the average pixel displacement are regarded as two key features for identifying the surface image state. The feature points of the salient areas of the surface are extracted and refined matching is performed, and the optical flow of the feature points of two adjacent frames of images is calculated.

[0085] (3) Based on kernel density estimation, the local density maximum of the number of feature points in the material surface image and the average optical flow vector data distribution is obtained, and used as the initial clustering center of the Gaussian mixture model (GMM). Based on the GMM model, the optical flow of the image feature points is clustered to obtain the state of the material surface.

[0086] (4) The material cycle is divided according to the changes in the density of feature points in the material surface image, and the key frames in the videos of different material cycles are extracted based on the results of state recognition.

[0087] The specific implementation plan is as follows:

[0088] (1) Based on the boundary prior and image feature space, the salient areas of the material surface image are located to reduce the influence of non-material surface areas and gas flow changes in the material surface image.

[0089] The material surface image includes non-material surface areas such as gas flow area, furnace wall area and material surface area composed of furnace charge, among which the material surface area is the area of ​​interest. When processing the material surface image, the non-material surface area will affect the accuracy of the processing result. In the blast furnace smelting process, the smaller the material surface morphology changes, the more stable the furnace is. Under normal operating conditions, the material surface areas with obvious changes in the central gas flow edge contour and the material surface areas with obvious brightness and texture in the image are more obvious in the material surface image. These areas have the most densely populated feature points, the largest number, and are most likely to attract people's visual attention. They are defined as the visually significant areas of the material surface image. In order to reduce the impact of non-material surface areas and gas flow changes on later image processing, the present invention extracts the significant areas of the material surface image.

[0090] For the material surface image, the furnace wall area, gas flow area and material surface area with blurred details and dim brightness can be regarded as the background of the image, and the place where the background and non-background area are connected is regarded as the boundary. The present invention introduces boundary prior to obtain a preliminary saliency map. In order to improve the efficiency of image salient area extraction and avoid the problem of long processing time caused by excessive image resolution, pixels with similar features are clustered into the same superpixel block through SLIC.

[0091] B={b1,b2,...,b n} (1)

[0092] Where B represents a set of superpixel blocks, and b represents each superpixel block.

[0093] Boundary connectivity is used to characterize the degree of connection between different pixel blocks and the boundary of the image. The greater the boundary connectivity, the smaller the probability of belonging to a salient area. The boundary connectivity can be expressed by the length of the target area at the boundary of the image and the square root of the area of ​​the area. The boundary connectivity of a superpixel block is expressed as follows:

[0094]

[0095] Where L(b m ) represents the number of pixels of the superpixel block on the boundary, A(b m ) represents the number of pixels in the entire superpixel block.

[0096] According to the boundary connectivity, the image pixel block b can be obtained m The probability of belonging to the background is as follows:

[0097]

[0098] Where P(b m ) represents the probability that the superpixel belongs to the background, σ b Indicates parameter, set to 1.

[0099] According to formula (3), the saliency values ​​of different pixel blocks in the material surface image can be obtained, but the obtained saliency values ​​are discontinuous, and the saliency values ​​of pixels close to the gas flow area are also relatively low. In order to improve the accuracy of saliency region extraction, the pixel saliency values ​​are corrected based on the material surface image feature space. The feature space of the material surface image includes color features, texture features, brightness features and space features. The overall difference between pixels is defined as follows:

[0100]

[0101] Where d1(p i ,p j )、d2(p i ,p j ) and d3(p i ,p j ) represent the pixels p of the surface image respectively. i and pixel p j The Euclidean distance of color feature, brightness feature and texture feature, d s (p i ,p j ) represents pixel p i and pixel p j The Euclidean distance of the spatial features between them, α represents a parameter, which is set to 3. Formula (4) can be understood as the significant region is the region where the brightness feature, texture feature, and color feature are greatly different and the position is relatively concentrated.

[0102] Based on the boundary prior and the overall difference in feature space, the saliency value of pixel i can be expressed as:

[0103]

[0104] Where S pi Represents pixel p i The significant value of d(p i ,p n ) represents pixel p i and pixel p n The overall difference, b pi Indicates that it contains pixel p i The superpixel block, N represents the pixel p i The number of pixels in the 8×8 neighborhood centered at pi ) represents the super pixel block b pi The probability of belonging to the background, d(p i ,p n ) is larger, S pi The bigger.

[0105] After obtaining the saliency values ​​of different pixel points of the material surface image, the OTSU method is used to perform binarization processing on the material surface image, and the material surface image is divided into the gas flow area and the non-gas flow area. After obtaining the position of the gas flow area, the saliency value of the gas flow area is reduced. The material surface image is binarized according to the calculated saliency value to obtain the final saliency area.

[0106] (2) The density of feature points and the average pixel displacement are regarded as two key features for identifying the surface image state. The feature points of the salient areas of the surface are extracted and refined matching is performed, and the optical flow of the feature points of two adjacent frames of images is calculated.

[0107] The density of feature points and pixel displacement of the material surface image have a strong correlation with the motion state of the blast furnace material surface. Generally speaking, the denser the feature points of the material surface image, the smaller the pixel displacement, and the slower the material surface falls, while the sparser the image feature points, the larger the pixel displacement, the faster the material surface falls, and the more unstable the material surface. For this reason, the embodiment of the present invention selects the density of feature points and the pixel displacement of the material surface image motion as two key features for the material surface video key frame extraction, where the density of feature points is represented by the number of feature points, and the pixel displacement of the image motion is represented by the average optical flow vector.

[0108] In order to reduce the impact of scale space on feature extraction, the surface image is downsampled to obtain surface images of different sizes. The surface images of different sizes are arranged from large to small, and Gaussian difference pyramid (DOG) is obtained through Gaussian convolution and Gaussian difference.

[0109] L(x,y,σ)=G(x,y,σ)*I(x,y) (6)

[0110]

[0111] Where I(x,y) represents the original image, σ represents the spatial scale, k represents the scale factor, G(x,y,σ) represents a scale-variable two-dimensional Gaussian function, and L(x,y,σ) represents the scale space of the image.

[0112] Each pixel in the DOG image is compared with its adjacent pixels and the pixels in the adjacent areas of the upper and lower images to find extreme points, which constitute a discrete set of feature point candidates.

[0113] After obtaining the discrete extreme points, the discrete extreme points are fitted into continuous extreme points through the Taylor formula, the unstable points are eliminated, and the edge influence is eliminated based on the Harris corner detection algorithm to obtain a more accurate feature point set L.

[0114] Calculate the scale value corresponding to the feature point and the angle and amplitude of the material surface image in the 16×16 neighborhood. Use the histogram to count the gradient and amplitude corresponding to the pixels in the area. The direction with the highest amplitude is the main direction, and the descriptor of the feature point is obtained.

[0115] Based on the Brute-Froce matching method, the feature point sets L and L′ of the material surface images of adjacent frames are roughly matched, and the distance between the descriptor of a feature point and the descriptor of other feature points is calculated. The point with the closest distance is used as a rough matching point of the feature point. Since the features of the material surface image are relatively simple, it is difficult to accurately identify the corresponding relationship of the feature points of the image pair, which will cause mismatching. Therefore, the mismatching points are eliminated based on the RANSAC algorithm to obtain the final matching result, such as Figure 2 As shown. At this time, the feature point set of two adjacent frames of images is expressed as:

[0116] P={P1,P2,P3,...,P n} (8)

[0117] P′={P′1,P′2,P′3,...,P′ n} (9)

[0118] After obtaining the feature point sets P and P′, the embodiment of the present invention uses the optical flow method to calculate the optical flow of all feature points. Assume that a feature point P of the t-th frame image in the surface video i (x, y), the motion components of the feature point in the horizontal and vertical directions are u x and v y , on the t+1 frame image, P i (x,y) moves to P′ i (x+△x,y+△y), feature point P iThe optical flow vectors in the horizontal and vertical directions of (x, y) are calculated as:

[0119] (u x ,v y )=(x+△xx,y+△yy)=(△x,△y) (10)

[0120] Where x and y represent the coordinates of the feature points.

[0121] After obtaining the optical flow vectors of all feature points in the image, considering that the material surface is relatively stable under normal working conditions and the material surface moves as a whole, in order to simplify the calculation, the average feature point optical flow vector is used as the moving pixel displacement of the material surface at a certain moment.

[0122]

[0123] Among them, m represents the number of image feature points, u a ,v a They represent the horizontal and vertical components of the pixel displacement of the image feature point a respectively.

[0124] (3) Based on kernel density estimation, the number of feature points in the surface image and the local density maximum of the average optical flow vector data distribution are obtained.

[0125] After obtaining the density of feature points in the salient area of ​​the material surface image and the pixel displacement of the feature points, the state of the material surface is determined based on these two features. Considering that the Gaussian mixture model can fit distributions of any shape, the traditional Gaussian mixture model (GMM) is very sensitive to the selection of initial values ​​and is prone to fall into local optimality due to poor selection of initial values. In order to accurately identify the state of the material surface, the present invention proposes a material surface state clustering method that integrates the local density maximum and the GMM model. First, the point with the largest local density is found according to the data distribution, and used as the initial value of the GMM model input, and then the GMM parameters are updated. The specific steps are as follows:

[0126] Step 1: Collect material surface images at different periods and in different states, and calculate the density of feature points (number of feature points) and average pixel displacement (average optical flow vector) between two adjacent frames.

[0127] Step 2: Use equal confidence probability to eliminate outliers and normalize the number of feature points and pixel displacement vectors.

[0128]

[0129] Among them, w represents the Chauville coefficient, d0 represents the outlier point, and d * ,d i ,d min ,d maxThey represent the normalized data, the data before normalization, the minimum value and the maximum value of the data respectively.

[0130] Step 3: After data preprocessing, the kernel density estimation function is used to obtain the probability density map of the data distribution. The kernel density estimation function is as follows:

[0131]

[0132] Where h is the smoothing parameter, K h (·) represents the kernel function, d represents the observed value, and f h (d) represents the probability density function, M represents the total number of samples, and d i Represents the i-th sample in the feature space sample set.

[0133] Step 4: The four points with the largest local density are taken as the initial cluster centers. The initial values ​​of the GMM model parameters are set according to the initial values. At the same time, the EM algorithm is used to update the GMM model parameters until the log-likelihood function reaches the maximum value and the parameter update is completed.

[0134]

[0135]

[0136] Among them, α i represents the probability that the current frame belongs to category i, μ i ,∑ i Respectively represent the mean and covariance matrix of the i-th Gaussian distribution, f(α i ,μ i ,∑ i ) represents the objective function of parameter update, L(α i ,μ i ,∑ i ) represents the log-likelihood function, m represents the number of samples, k represents the number of Gaussian functions, x j represents the current sample, P(x j / m i ) represents the sample x j The probability of belonging to the i-th Gaussian distribution;

[0137] Step 5: Draw a Gaussian distribution curve and classify the data within or close to the Gaussian distribution curve into the same state.

[0138] (4) The material cycle is divided according to the changes in the density of feature points in the material surface image, and the key frames in the videos of different material cycles are extracted based on the results of state recognition.

[0139] The state of the material surface is the main basis for judging whether the current frame is a key frame. According to different requirements, a set of key frames that meet different conditions can be obtained. Figure 3 As shown in the figure, since the top material distribution is periodic and intermittent, the morphology and clarity of the material surface in different periods will be different. Therefore, it is necessary to divide the material surface video into video segments according to the distribution cycle, obtain material surface videos of different periods, and then extract key frames from the videos of different periods. Figure 4 The specific steps of extracting the key frames of the surface video in this embodiment are as follows:

[0140] Step 1: Input the surface video and initialize related parameters (current frame t, number of frames N0, period T0).

[0141] Step 2: Considering that the material surface is relatively clear during the intermittent material distribution, the image feature points are dense; in the material distribution process, due to the occlusion of furnace charge and dust, the effective information of the material surface is reduced, and the image feature points are sparse. Therefore, the material distribution cycle can be divided according to the change in the density of image feature points. In order to reduce the random error of the feature point number statistics, the average number of feature points of the material surface image per second is calculated and normalized to [0,1], which is used to characterize the density of feature points of the material surface at the current moment f t , and the density of feature points of the surface image of the previous second f t-1 For comparison, if f t -f t-1 ≤T, it is considered to be the same period, T0 remains unchanged; if f t -f t-1 >T, and then determine whether f is satisfied in the next few seconds. t+i -f t+i-1 >0(i=1,2,3), if it is satisfied, it is considered that the material surface is gradually becoming clear, one-time cloth making has been completed, and it enters the next cycle, that is, T0+1; at the same time, the frame number is initialized to 1, that is, N0=1.

[0142] Step 3: Calculate the Euclidean distance from different cluster centers to obtain the state of the current frame. If it is in a fast sinking state or an unstable state, it is judged as a non-key frame and the frame number N0 remains unchanged; if it is in a stable state or a slow sinking state, it is judged as a candidate key frame and the frame number N0 increases by 1.

[0143] Step 4: Determine the sampling frequency based on the number of candidate key frames to ensure that the number of key frames extracted in each cycle is the same. From the candidate key frames, the image frame corresponding to the cluster center, the image frame with the largest number of feature points and the smallest optical flow vector, and the image obtained at a fixed sampling frequency are selected as key frames to obtain a set of key frames for a cycle.

[0144] Step 5: Repeat Step 2 to Step 4 to obtain key frames of different cloth cycles in the material surface video and obtain the final key frame set. It is worth noting that the key frame determination rule can be changed according to actual needs so that the extracted key frames can meet the conditions.

[0145] The method for automatically extracting key frames of blast furnace material surface video provided by the embodiment of the present invention uses a high-temperature industrial endoscope to collect material surface video of the blast furnace smelting process, locates the significant area of ​​the material surface image based on boundary prior and image feature space, extracts accurate feature points of the significant area of ​​the image, and calculates the pixel displacement of two adjacent frames of images according to the optical flow method; then a method of fusing local density maximum and GMM model clustering is proposed to identify the material surface state; finally, the video collected by the high-temperature industrial endoscope is divided into different periods according to the change in the density of image feature points, and the key frames of each material distribution period are extracted based on the material surface state recognition result. The present invention can accurately identify the material surface state and eliminate redundant information of the material surface video, and automatically extract the material surface video key frames with stable central airflow and obvious image features from the blast furnace material surface video. At the same time, the present invention is suitable for the automatic extraction of key frames of other videos with periodic changes and uneven image quality.

[0146] Furthermore, the method for automatically extracting key frames of blast furnace material surface video proposed in the embodiment of the present invention can automatically extract key frames with stable central airflow and obvious features from blast furnace material surface videos with periodic morphological changes and non-uniform image quality, which can provide more accurate and reliable information for on-site operators to grasp the operation status in the furnace, changes in material surface morphology and distribution of coal gas flow in a timely manner, and greatly increase the accuracy and efficiency of later processing such as deep extraction based on material surface images. First, the material surface video during the operation of the blast furnace is collected based on a high-temperature industrial endoscope, and the material surface video is analyzed to divide the material surface into four states: unstable, static, slow sinking and fast sinking. In order to eliminate redundant information of the video, the optical flow of the feature points in the significant area of ​​the material surface image is calculated, and the fusion of the local density maximum and the GMM clustering method is proposed to identify the state of the material surface. According to the change in the number of feature points and the state of the material surface, the key frames with stable central airflow and obvious image features in different material distribution cycles are automatically distinguished. The embodiment of the present invention is suitable for the extraction of key frames of videos with periodic changes, multiple states and uneven image quality, and can extract key frames that meet the requirements according to different actual needs.

[0147] Embodiment 3

[0148] This example uses a 2650m 3The blast furnace is used as the experimental platform. The material surface video of the blast furnace smelting process is collected by a high-temperature industrial endoscope inserted into the furnace. The key frame automatic extraction method proposed in the present invention is used to automatically extract the key frames with stable central airflow and obvious characteristics from a large number of videos with periodic changes and uneven quality. The specific implementation steps are as follows:

[0149] 1. A large number of material surface videos of blast furnace smelting processes in different cycles were collected based on high-temperature industrial endoscopes.

[0150] 2. In order to reduce the impact of non-surface areas and gas flow changes in the image, the salient areas of the image are extracted based on boundary priors and image feature space.

[0151] 3. The feature points of the salient areas of the image are extracted and matched in a refined manner to obtain a set of feature points that can correspond one-to-one between two adjacent frames of images, and the optical flow values ​​of the feature points are calculated.

[0152] 4. The material surface images at different periods and states were collected, and the density of feature points and the average pixel displacement were counted respectively. The density of feature points is expressed by the number of feature points, and the average pixel displacement is expressed by the average optical flow vector of feature points.

[0153] 5. The number of feature points and the local density maximum of the average optical flow vector are obtained based on kernel density estimation. The density of feature points and the average pixel displacement are clustered based on the Gaussian mixture model, where the initial cluster center is the data corresponding to the local density maximum.

[0154] 6. Calculate the density of feature points of the material surface image per second, and compare the changes in the density of feature points of adjacent material surface images per second. If the density of feature points increases suddenly and continues to increase at the next moment, it is considered that a fabric cycle ends and the next fabric cycle begins.

[0155] 7. The images in the fast sinking and unstable state are judged as non-key frames, and the material surface images in the static and slow sinking state are selected as candidate key frames. The sampling frequency is determined according to the number of candidate key frames in the same period, and the cluster center and the material surface image obtained by fixed sampling frequency are extracted from the candidate key frames and output as key frames. The extracted key frame center has stable airflow and obvious characteristics.

[0156] Reference Figure 5 The blast furnace charge surface video key frame automatic extraction system proposed in the embodiment of the present invention comprises:

[0157] A memory 10, a processor 20, and a computer program stored in the memory 10 and executable on the processor 20, wherein the processor 20 implements the steps of the method for automatically extracting key frames from a blast furnace material surface video proposed in this embodiment when executing the computer program.

[0158] The specific working process and working principle of the automatic key frame extraction system of the blast furnace material surface video of the present embodiment can refer to the working process and working principle of the automatic key frame extraction method of the blast furnace material surface video of the present embodiment.

[0159] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for automatically extracting key frames from a blast furnace charge surface video, characterized in that: The method comprises: Obtaining a significant region of the material surface image, wherein obtaining the significant region of the material surface image includes: The surface image is divided into superpixel blocks with a preset threshold through SLIC clustering; Calculate the boundary connectivity between superpixel blocks and boundaries; According to the boundary connectivity, the probability that the superpixel block belongs to the background is obtained; According to the probability that the super pixel block belongs to the background, the saliency value of the super pixel block is obtained; Obtaining a significant region of the surface image according to the significant value of the superpixel block, wherein obtaining a significant region of the surface image according to the significant value of the superpixel block includes: Based on the feature space of the material surface image, the overall difference between the pixels of the material surface image is obtained, and the calculation formula for obtaining the overall difference between the pixels of the material surface image is: Among them, d1(p i ,p j )、d2(p i ,p j ) and d3(p i ,p j ) represent the pixels p of the surface image respectively. i and pixel p j The Euclidean distance of color feature, brightness feature and texture feature, d s (p i ,p j ) represents pixel p i and pixel p j The Euclidean distance of the spatial features between them, α represents the parameter, which is set to 3; According to the saliency value of the superpixel block and the overall difference between pixels, the saliency value of the pixel is obtained, wherein the calculation formula for obtaining the saliency value of the pixel is: in, Represents pixel p i The significant value of d(p i ,p n ) represents pixel p i and pixel p n The overall difference of Indicates that it contains pixel p i The superpixel block, N represents the pixel p i The number of pixels in the 8×8 neighborhood centered at Represents a superpixel block Probability of belonging to the background; Binarization is performed on the material surface image according to the significant value of the pixel to obtain the significant area of ​​the material surface image; Extract feature points of salient areas; Match the feature points of two adjacent frames of material surface images, and calculate the number of feature points and the optical flow of feature points of the two adjacent frames of material surface images; Based on kernel density estimation, the local density maximum of the number of feature points and the average optical flow vector data distribution is obtained to obtain the initial clustering center of the Gaussian mixture model, wherein the average optical flow vector is the average value of the optical flow of all feature points in the material surface image; Based on the initial clustering center, the Gaussian mixture model is used to cluster the optical flow of the feature points of the material surface image to obtain the state of the material surface, where the state of the material surface includes fast sinking state, unstable state, stable state and slow sinking state; The fabric cycle is divided according to the density change of the feature points of the material surface image, and the key frames in the videos of different fabric cycles are extracted based on the state of the material surface.

2. The method for automatically extracting key frames from a blast furnace charge surface video according to claim 1, characterized in that: The feature points of the salient area are extracted as follows: The regional images corresponding to the salient regions are downsampled to obtain regional images of different sizes, and the regional images of different sizes are arranged from large to small, and Gaussian difference pyramids are obtained through Gaussian convolution and Gaussian difference. Compare each pixel of the image in the Gaussian difference pyramid with the adjacent pixels and the pixels in the adjacent areas of the upper and lower layers of the image to find the extreme points, thereby obtaining discrete extreme points; The discrete extreme points are fitted into continuous extreme points through the Taylor formula, the unstable points are eliminated, and the edge influence is eliminated based on the Harris corner detection algorithm to obtain a continuous set of feature points, thereby obtaining the feature points of the significant area.

3. The method for automatically extracting key frames from a blast furnace charge surface video according to claim 2, characterized in that: The calculation formula for calculating the number of feature points and the optical flow of feature points of two adjacent frames of material surface images is: (u x ,v y )=(x+Δx-x,y+Δy-y)=(Δx,Δy), Among them, u x and v y Respectively represent the i-th feature point P of the t-th frame surface image i (x, y) moves to the i-th feature point P of the t+1-th frame surface image i ′(x+Δx,y+Δy) is the motion component in the horizontal and vertical directions, where x and y represent the feature point P i The coordinates of (x, y), x+Δx, y+Δy represent the feature point P i The coordinates of ′(x+Δx,y+Δy).

4. The method for automatically extracting key frames from a blast furnace charge surface video according to any one of claims 1 to 3, characterized in that: Based on kernel density estimation, the number of feature points and the local density maximum of the average optical flow vector data distribution are obtained, and the initial clustering center of the Gaussian mixture model is obtained, including: Collect material surface images at different periods and in different states, and calculate the number of feature points and the average optical flow vector of two adjacent frames of images, wherein the average optical flow vector is the average value of the optical flow of all feature points in the material surface image; Equal confidence probability is used to remove outliers, and the number of feature points and the average optical flow vector are normalized; The kernel density estimation function is used to obtain the probability density map of the number of feature points and the average optical flow vector data distribution, where the kernel density estimation function is specifically: Among them, h represents the smoothing parameter, K h (·) represents the kernel function, d represents the observed value, and f h (d) represents the probability density function, M represents the total number of samples, and d i Represents the i-th sample in the feature space sample set; According to the probability density map, the four points with the largest local density are obtained as the initial cluster centers, and they are used as the initial cluster centers of the Gaussian mixture model.

5. The method for automatically extracting key frames from a blast furnace charge surface video according to claim 4, characterized in that: Based on the initial cluster center, the Gaussian mixture model is used to cluster the optical flow of the feature points of the material surface image to obtain the state of the material surface, including: The EM algorithm is used to update the parameters of the Gaussian mixture model until the log-likelihood function reaches its maximum value, and the parameter update ends. The log-likelihood function is specifically: Among them, α i represents the probability that the current frame belongs to category i, μ i ,∑ i Respectively represent the mean and covariance matrix of the i-th Gaussian distribution, f(α i ,μ i ,∑ i ) represents the objective function of parameter update, L(α i ,μ i ,∑ i ) represents the log-likelihood function, m represents the number of samples, k represents the number of Gaussian functions, x j represents the current sample, P(x j / μ i ) represents the sample x j The probability of belonging to the i-th Gaussian distribution; Draw a Gaussian distribution curve and classify the data within or close to the Gaussian distribution curve into the same state.

6. The method for automatically extracting key frames from a blast furnace charge surface video according to claim 5, characterized in that: The cloth cycle is divided according to the density change of the feature points of the material surface image, and the key frames in the video of different cloth cycles are extracted based on the state of the material surface, including: The density of feature points is obtained according to the average number of feature points of the material surface image per second; Compare the density of feature points of the material surface image at the current moment with the density of feature points of the material surface image at the previous second, and divide the material distribution cycle according to a preset comparison threshold; Calculate the Euclidean distance between the feature value of the current frame and different cluster centers to obtain the state of the current frame. If it is in a fast sinking state or an unstable state, it is determined to be a non-key frame and the frame number remains unchanged. If it is in a stable state or a slow sinking state, it is determined to be a candidate key frame and the frame number is increased by 1. The sampling frequency is determined according to the number of candidate key frames to ensure that the number of key frames extracted in each cycle is the same. From the candidate key frames, the image frame corresponding to the cluster center, the image frame with the largest number of feature points and the smallest feature point optical flow, and the image obtained at a fixed sampling frequency are selected as key frames to obtain a set of key frames for a cycle. The key frame sets of different material distribution cycles are extracted to obtain the blast furnace material surface video frame set.

7. A blast furnace charge surface video key frame automatic extraction system, the system comprising: A memory (10), a processor (20), and a computer program stored in the memory (10) and executable on the processor (20), characterized in that the processor (20) implements the steps of any of the methods described in claims 1 to 6 when executing the computer program.

Images