Time sequence optical flow feature anomaly detection method, device and system for mine underflow pump station and medium

Abstract

The invention discloses an anomaly detection method based on time sequence optical flow characteristics for a mine underflow pump station. A fixed camera video is accessed to the system, dense optical streams are calculated according to frames, inter-frame optical streams are aggregated according to seconds in a minute-level judgment time window, and a motion mask is constructed according to a quantile threshold value; features such as activity coverage, high intensity quantile, direction uniformity, gravity consistency and positive divergence are extracted in the mask, mean value / maximum equal pooling is carried out in a time window, and a window-level feature vector is formed; inputting the vector into a lightweight three-layer feed-forward shared embedder, training a class of learning models only by using normal samples to obtain an abnormal score and a threshold value, and outputting a water spraying and dripping probability on the same embedder. According to the method, a large number of abnormal samples and fixed threshold values are not needed, and stable recognition can be kept in multiple working conditions, day and night and camera position micro-change scenes; compared with the prior art, the method has the advantages of early warning, low false alarm, easy deployment, real-time edge and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to industrial video monitoring and computer vision, in particular to a time-series optical flow feature-based anomaly detection method, device, system and medium for a mine bottom flow pump station. BACKGROUND

[0002] The mine bottom flow pump station is mostly of well type or semi-closed structure, and is provided with a water collecting tank and a pump set at the bottom. Once water spraying / splashing leakage occurs, the excessive water inflow in a short time is easy to exceed the collection and discharge capacity, so that the pump house is quickly flooded and the motor base, coupling, on-site instrument box and cable trench are submerged, thereby causing short circuit, tripping, shutdown and even equipment damage, and there is a risk of personnel safety. The on-site environment is long-term high humidity and high dust, and the easy leakage points are concentrated in the flange connection, valve stuffing box, pump shaft seal and ground water collection ditch periphery; the abnormal form has locality and intermittence: the water dripping is mostly small-range continuous falling along the gravity direction, the leakage on the ground is manifested as slow expansion of the connected domain, and the water spraying is manifested as strong movement of short-time burst, direction divergence and area sudden increase.

[0003] For the above-mentioned abnormality, the two common methods at the present stage have the following deficiencies: one is that the process interlocking (flow, pressure, liquid level) usually needs to act when the liquid level rises significantly, and lacks direct perception of visible abnormalities occurring on the surface of the equipment; the other is that the common video methods such as frame difference and fixed threshold are easy to cause false positives and false negatives under complex environmental disturbances, and it is also difficult to effectively distinguish between water dripping, water spraying and other abnormal conditions. In addition, the general learning method (deep model with strong supervision) usually relies on a large number of and balanced abnormal samples and fine labeling, and the bottom flow pump station abnormality is already scarce, which is difficult to meet the data and labeling requirements. SUMMARY

[0004] The embodiment of the present application provides a time-series optical flow anomaly detection method for a mine bottom flow pump station, which is used to solve the technical problems that the existing mine bottom flow pump station cannot early perceive and is easy to cause false positives, false negatives and confusion in detecting water spraying / splashing leakage abnormality.

[0005] According to one aspect of the present application, a time-series optical flow feature-based anomaly detection method for a mine bottom flow pump station is provided, comprising: obtaining real-time video frames; calculating a dense optical flow vector field for adjacent frame pairs; aggregating the dense optical flow vector fields of all frame pairs corresponding to the minute-level determination time window by seconds, and constructing a motion mask with a quantile threshold; extracting activity coverage, intensity high quantile, direction uniformity, gravity consistency and divergence features in the mask, and pooling in the time window to form a window-level feature vector; The feature vector is input into a lightweight three-layer feedforward shared embedder to train a one-class learning model with normal samples to obtain an abnormal score and a threshold, and to output a water spray and water drop probability on the same embedding.

[0006] In a second aspect of the present application, an abnormality detection device based on time sequence optical flow features for a mine bottom flow pump station is provided, comprising: a frame acquisition module configured to acquire real-time video frames; a dense optical flow vector field calculation module configured to calculate a dense optical flow vector field for each adjacent frame pair; an aggregation module configured to aggregate the dense optical flow vector fields of all frame pairs in a minute-level determination time window by seconds, and construct a motion mask with a quantile threshold; a feature vector formation module configured to extract activity coverage, intensity high quantile, direction uniformity, gravity consistency and divergence features in the mask, and pool them in the time window to form a window-level feature vector; an embedding module configured to input the feature vector into a lightweight three-layer feedforward shared embedder to train a one-class learning model with normal samples to obtain an abnormal score and a threshold, and output a water spray and water drop probability on the same embedding.

[0007] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method of the first aspect of the present application.

[0008] In a fourth aspect of the present application, a computer readable storage medium is provided for storing a computer program, which when executed on a computer, causes the computer to execute the method of the first aspect of the present application.

[0009] Advantages: The present application proposes a time sequence optical flow abnormality detection method for a mine bottom flow pump station. Without the need for a large number of abnormal samples, the method calculates a dense optical flow for all frame optical flows in a minute-level time window, constructs a second-level motion mask and extracts coverage, intensity, direction uniformity, gravity consistency and positive divergence features, trains a one-class learning model with normal samples to form an adaptive discrimination boundary, and distinguishes between water spray and water drop in abnormalities; without the need for a large number of abnormal samples, early and stable identification of two types of exudation is achieved, the generalization ability under multiple working conditions is improved, false positives and false negatives are reduced, the timeliness and reliability of early warning are enhanced, and the operation safety and operation efficiency of the bottom flow pump station are improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their Figure 1 is a flow chart of a time sequence optical flow anomaly detection method for a mine bottom flow pump station according to an embodiment of the application.

[0011] Figure 2 is a schematic diagram of a time sequence optical flow anomaly detection device for a mine bottom flow pump station according to an embodiment of the application.

[0012] Figure 3 is a schematic diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0013] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0014] It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flow chart, in some cases, the steps shown or described herein can be executed in a different order.

[0015] In the bottom flow pump station scene, the challenge comes from how to stably identify the weak but possibly rapidly evolving external leakage behavior in the complex background of high humidity, high dust, light fluctuation, etc. Most existing video methods rely on single frame or short time frame difference, which can only capture instantaneous changes and cannot depict the persistence structure of dripping water and spraying water in multiple frame time scales. While deep learning classification models rely on a large number of balanced abnormal samples, and it is almost impossible to establish an effective model in the pump station environment where anomalies are extremely scarce. Therefore, the existing technical path generally stays at the level of "whether there is motion", lacks the idea of extracting stable evidence from long time segments, and it is difficult to form a reliable judgment on the subdivided external leakage behaviors such as dripping water and spraying water.

[0016] As Figure 1 is a flow chart of an anomaly detection method based on time sequence optical flow features for a mine bottom flow pump station according to an embodiment of the application, as shown in the figure, the method comprises: Step S102, acquiring real-time video frames.

[0017] Step S104, calculating dense optical flow vector fields according to adjacent frame pairs.

[0018] Step S106, aggregating the dense optical flow vector fields of all frame pairs corresponding to the minute-level determination time window by seconds, and constructing a motion mask with a quantile threshold.

[0019] Step S108, extract activity coverage, intensity high quantile, direction uniformity, gravity consistency and divergence features in the mask, and pool in the time window to form a window-level feature vector.

[0020] Step S110, input the feature vector into a lightweight three-layer feedforward shared embedder to train a one-class learning model with normal samples to obtain an anomaly score and a threshold, and output a water spray and water drop probability on the same embedding.

[0021] The embodiment of the present application is conceived in view of the limitations of the prior art, and no longer relies on single-frame or short-time information, but introduces a minute-level time window to reorganize multiple frames of optical flow in the time dimension into a set of "time sequence motion evidence", and uses "minute-level time sequence optical flow features" to construct a generalizable anomaly judgment mechanism.

[0022] By analyzing the physical characteristics of the leakage phenomenon, it can be found that water drops exhibit weak but continuous, highly consistent direction; water spray exhibits instantaneous burst, direction divergence and significant local positive divergence; normal disturbance has the characteristics of short-time randomness and lack of continuity. Based on this, the method of the embodiment of the present application proposes a set of optical flow statistical features that can stably separate these behaviors within a time window, including coverage, intensity high quantile, direction uniformity, gravity consistency and positive divergence, so that the abnormal structure that is difficult to distinguish in a single frame naturally widens the gap in a statistical sense. The idea of converting "weak anomalies that are difficult to identify in spatial form" into "measurable statistical quantities in the time dimension".

[0023] Compared with the frame difference method, the method can significantly reduce false positives caused by light disturbance; compared with the single-frame optical flow threshold method, the method significantly improves the stability of water drop recognition through minute-level time sequence accumulation; compared with deep classification models that rely on a large number of abnormal samples, the method can still work normally under the condition that abnormal samples are extremely scarce.

[0024] The video frames in step S102 are obtained by a camera installed at the test site, which is aimed at a predetermined set of monitoring regions , and a sampling setup is performed, including setting a reference frame rate , resolution , and recording a timestamp for subsequent sliding determination. The above set of monitoring regions is stored in the form of a mask, and regions with an area below a threshold are not included in the calculation, and subsequent optical flow calculation and feature statistics are limited to .

[0025] In some preferred embodiments, in order to be applicable to different video frame rates, the setting method of the minute-level time window T includes: when the reference frame rate , T takes 1-3 minutes; when T takes 3-5 minutes; when T takes 5-10 minutes. That is, the greater the reference frame rate, the more frames per minute, and the shorter the corresponding time window. At the end of each T, the optical flow of the video data in this time period is calculated and the timing optical flow features are summarized, completing an anomaly determination and outputting the result.

[0026] In some preferred embodiments, in order to quickly capture possible anomalies, the step S104 detects changes between adjacent frames from frame pairs. Specifically, the step S104 calculates a dense optical flow vector field from adjacent frame pairs, including: Step S1041, selecting two consecutive frames from the continuous frame sequence corresponding to the video frame to form a frame pair , only the monitoring area covered range participates in subsequent statistics; Step S1042, using a deep learning dense optical flow network to perform end-to-end inference on each frame in the frame pair, including inputting the images of the monitoring area of the two frames, and outputting the pixel-level displacement field between the two frames .

[0027] In the above process, the input resolution is scaled and padded as required during inference to maintain the optical flow result, which can be restored to the original image scale according to the coordinates. The deep learning optical flow model can realize real-time inference at the target resolution and frame rate through hardware acceleration, meeting the timeliness requirements of the method for periodic determination in the time window T.

[0028] In some preferred embodiments, in the step S106, the dense optical flow is calculated for all adjacent frame pairs in the minute-level determination time window T minutes, and the dense optical flow vector fields of all frame pairs are aggregated by seconds, including: According to the timestamp, the frame pairs are classified by seconds. In order to preserve the burst peak and obtain a robust direction field, the dense optical flow vector fields of all adjacent frame pairs in the determination time window T minutes are aggregated by seconds in the pixel coordinates , where the second-level aggregation of the frame pair set of the t-th second is as follows: where, is the optical flow amplitude of the frame pair p; is the short-time burst peak in the pixel coordinates in the t-th second, obtained by taking the maximum amplitude in the second; is the robust direction field in the pixel coordinates A weighted average of the two directional displacements, by vector magnitude, to suppress noise; The unit directional field for the t-th second.

[0029] In some preferred embodiments, in the step S106, the motion mask is constructed with quantile threshold, comprising: In each monitoring region , an adaptive threshold is constructed based on quantile, and the motion mask for the t-th second is formed as follows: wherein, represents the p-th quantile operator; is an indicator function; is a binary mask, representing the set of pixels with significant motion in during the t-th second; is the number of pixels in the binary mask in during the t-th second.

[0030] In some preferred embodiments, in the step S108, the activity coverage, intensity high quantile, direction uniformity, gravity consistency and divergence are extracted in the mask, and are pooled in the time window to form the window-level feature vector, comprising: The activity coverage wherein, is the number of ROI pixels, represents the proportion of spatial coverage with significant motion; used to depict the feature that the coverage is significantly increased when water is sprayed; The intensity high quantile wherein, is the 95-th quantile of the pixel amplitude above the threshold; reflects the upper bound of motion intensity; used to depict the high-momentum spray of water spray, which is usually significantly higher than water leakage; The direction uniformity wherein, is the synthetic norm of the unit directional vector, the larger the value, the more single the direction, and the water spray with multi-directional splashing leads to , while the water leakage / dripping along a single downward direction makes ; The gravity consistency , is the unit vector of the vertical downward gravity, represents the average consistency with the direction of gravity; used for water leakage / dripping identification ( high), and combined with the direction uniformity to distinguish the water spray not dominated by gravity ( lower); The divergence index , wherein,

[0031] Mainly used for water spray discrimination, while Water leakage / dripping as the main criterion (also can be combined with To ensure consistency across the machine and across ROI, actual deployment can be based on normal data on Each dimension is standardized.

[0032] In some preferred embodiments, in step S110, the vector is input into a lightweight three-layer feedforward shared embedding to train a one-class learning model with normal samples to obtain an anomaly score and a threshold, and output a water spray and water dripping probability on the same embedding, comprising: Wherein, Wherein, .

[0033] In order to give priority to water spray, the present scheme adopts a lower water spray trigger threshold and gives a higher weight in the joint target. If the two probabilities are close and high at the same time, the mutual exclusion term in the training period has reduced overlap; still conflict, take the larger one and refer to the physical evidence: With Higher tendency to water spray, With Higher tendency to water dripping.

[0034] Input dimension Equal to the dimension of the window vector.

[0035] The model runs online in the order of decoding->optical flow->feature->standardization->embedding->two event probability, and outputs once at the end of the time window And the final decision.

[0036] The method of the embodiment of the application seizes the particularity that the optical flow is susceptible to illumination fluctuation and steam interference in the pump station environment, leading to unstable inter-frame estimation, and converts the noisy optical flow into stable time series statistics through strategies such as second-level maximum amplitude aggregation, weighted vector average and quantile gating, thereby significantly enhancing the availability of the optical flow. Moreover, considering that abnormal leakage is extremely rare, the application establishes an adaptive discrimination boundary by relying only on normal samples, thereby avoiding the dependence of a strong supervision model on the number of abnormal samples. Furthermore, the application introduces a lightweight linear discrimination structure on a shared embedding space, and jointly optimizes a small number of labeled segments and a large number of unlabeled samples, thereby realizing further subdivision of the abnormality and overcoming the defect that the traditional method cannot distinguish between water spraying and water dripping in the abnormality. Through the above technical breakthrough, the embodiment of the application can realize early, stable and low false alarm leakage detection in the pump station environment which lacks abnormal data, has complex background and frequent camera changes, thereby significantly improving the operation safety and reliability of the scene.

[0037] Based on the same concept as the above method embodiment, as shown in Figure 2 The embodiment of the application also proposes an abnormality detection device for a mine bottom flow pump station based on time series optical flow features, which comprises: a frame acquisition module for acquiring real-time video frames; a dense optical flow vector field calculation module for calculating a dense optical flow vector field according to adjacent frames; an aggregation module for aggregating the dense optical flow vector fields of all frame pairs in a minute-level determination time window by seconds, and constructing a motion mask with a quantile threshold; a feature vector forming module for extracting activity coverage, intensity high quantile, direction uniformity, gravity consistency and divergence features in the mask, and pooling in the time window to form a window-level feature vector; an embedding module for inputting the feature vector into a lightweight three-layer feedforward shared embedder to train a one-class learning model with normal samples to obtain an abnormality score and a threshold, and outputting a water spraying and water dripping probability on the same embedding.

[0038] It should be noted that although several units or sub-units of the device are mentioned in the above detailed description, such division is only exemplary and not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units.

[0039] Based on the same inventive concept as the method embodiments, the electronic device in the embodiments of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the electronic device implements the control method in the above embodiments.

[0040] In an embodiment, the electronic device can be a server, and in this embodiment, the structure of the electronic device can be as shown in the figure, including a memory 2001, a communication module 2003, and one or more processors 2002. Figure 3

[0041] The memory 2001 is used to store the computer program executed by the processor 2002. The memory 2001 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and programs required for running instant messaging functions, etc.; the data storage area can store various instant messaging information and operation instruction sets, etc.

[0042] The memory 2001 can be a volatile memory such as a random-access memory (RAM); the memory 2001 can also be a non-volatile memory such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 2001 can be any other medium capable of carrying or storing desired computer programs in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 2001 can be a combination of the above memories.

[0043] The processor 2002 can include one or more central processing units (CPUs) or digital processing units, etc. The processor 2002 is used to call the computer program stored in the memory 2001 to implement the above-mentioned audio data processing method.

[0044] The communication module 2003 is used to communicate with terminal devices and other servers.

[0045] The embodiments of the present application do not limit the specific connection medium between the above-mentioned memory 2001, communication module 2003 and processor 2002. In the embodiments of the present application, the memory 2001 and the processor 2002 are connected through a bus 2004, and the bus 2004 is connected through a bus interface 2005. Figure 3 Figure 3 ​​The connections between the other components are described using arrows, which are merely illustrative and not limiting. The bus 2004 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 1 Only one arrow is used to describe the connections between the components, but this does not mean that there is only one bus or only one type of bus.

[0046] With the same inventive concept as the above method embodiments, the embodiments of the present application also provide a computer readable storage medium for storing a computer program, which, when running on a computer, causes the electronic device to implement the control method in the above embodiments. The computer readable storage medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0047] With the same inventive concept as the above method embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program, when the program product is running on an electronic device, the computer program is used to make the electronic device execute the steps in the control method according to various exemplary embodiments of the present application described in the specification. The program product can adopt any combination of one or more readable media. These computer program commands can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the commands executed by the processor of the computer or other programmable data processing device produce a method for implementing the functions specified in one or more flows or blocks and / or blocks. Figure 1 The flow or the plurality of flows and / or the block Figure 1 The device for performing the functions specified in one or more flows or blocks and / or blocks.

[0048] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all changes and modifications falling within the scope of the present application.

Claims

1. An anomaly detection method based on temporal optical flow characteristics for mine underflow pumping stations, characterized in that, include: Acquire real-time video frames; Calculate the dense optical flow vector field based on adjacent frame pairs; Within a minute-level decision window, the dense optical flow vector fields of all corresponding frame pairs are aggregated by second, and a motion mask is constructed using a quantile threshold. The features of active coverage, high intensity quantile, orientation regularity, gravity consistency and divergence are extracted within the mask and pooled within the time window to form a window-level feature vector. The feature vector is input into a lightweight three-layer feedforward shared embedding unit, and a learning model is trained with normal samples to obtain anomaly scores and thresholds. The probability of water spraying and dripping is output on the same embedding.

2. The method according to claim 1, characterized in that, The method for setting the minute-level time window T includes: when the base frame rate When, T is taken for 1-3 minutes; when Take T for 3-5 minutes; when When taking T, take it for 5-10 minutes.

3. The method according to claim 1, characterized in that, The calculation of the dense optical flow vector field by adjacent frame pairs includes: From the consecutive frame sequence corresponding to the video frame Two consecutive frames are selected to form a frame pair; The two frames in the frame pair are used to perform end-to-end inference for each frame using a deep learning dense optical flow network, including the monitoring regions of the two frames. The image is used as input, and the pixel-level displacement field between the two frames is used as output. .

4. The method according to claim 3, characterized in that, Within the minute-level decision window, the dense optical flow vector fields of all corresponding frame pairs are aggregated second by second, including: The dense optical flow vector field of all adjacent frame pairs within the time window T minutes is timestamped at pixel coordinates. Perform second-level aggregation, where for the set of frame pairs at second t... The second-level aggregation is as follows: in, The optical flow amplitude of frame pair p; The pixel coordinates in the t-th second The short-duration jet peak on the surface; For the pixel coordinates in the t-th second Weighted average of displacements in two directions; Let be the unit direction field at second t.

5. The method according to claim 4, characterized in that, The method of constructing a motion mask using quantile thresholds includes: In each monitoring area Internally, based on The quantiles are used to construct an adaptive threshold, and a motion mask is formed at second t as follows: ,in, Represents a p-bit number operator; is the indicator function; is the binary mask, representing the content within the t-th second. A set of pixels with significant motion; Within the t-th second The number of pixels in the binary mask.

6. The method according to claim 5, characterized in that, The process of extracting activity coverage, intensity high quantile, orientation regularity, gravity consistency, and divergence features within the mask, and pooling them within the time window to form a window-level feature vector, includes: The activity coverage ,in, For the number of pixels in the ROI, Indicates the spatial coverage ratio of significant motion; The intensity high quantile ,in, The 95th percentile of the above-threshold pixel amplitude; The directional regularity ,in, The norm of the composition of unit vectors in the direction; The consistency of gravity , This is the vertically downward unit vector of gravity; The divergence index ,in, 。 7. The method according to claim 1, characterized in that, The step of inputting the vector into a lightweight three-layer feedforward shared embedding unit, training a class-specific learning model with normal samples to obtain anomaly scores and thresholds, and outputting the probability of water spraying and dripping on the same embedding includes: in, in, 。 8. An anomaly detection device based on time-series optical flow characteristics for a mine underflow pumping station, characterized in that, include: The frame acquisition module is used to acquire real-time video frames; The dense optical flow vector field calculation module is used to calculate the dense optical flow vector field on adjacent frame pairs; The aggregation module is used to aggregate the dense optical flow vector fields of all corresponding frame pairs by second within a minute-level decision time window, and construct a motion mask with a quantile threshold. The feature vector forming module is used to extract active coverage, intensity high quantile, orientation uniformity, gravity consistency and divergence features within the mask, and pool them within the time window to form window-level feature vectors; The embedding module is used to input the feature vector into a lightweight three-layer feedforward shared embedding unit, train a learning model with normal samples to obtain anomaly scores and thresholds, and output the probability of water spraying and dripping on the same embedding.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 7.

Images