Debris flow video recognition model sample set, model construction method and edge monitoring system

Abstract

The invention discloses a debris flow video recognition model sample set, a model construction method and an edge monitoring system. The invention provides a debris flow edge monitoring scheme aiming at the defects of high calculation overhead and dependence on hardware performance in the prior art. A sample generated by the sample set construction method of the debris flow video recognition model adopts a grey-scale map to represent frame image information of a monitoring video, sample data considers three objectives of containing disaster dynamics, removing environmental interference and simplifying image feature expression, and meets the requirements of recognition model training standard reaching and edge equipment performance adaptation. The debris flow video recognition model construction method provides an improved Tiny Darknet model for training. The debris flow edge monitoring system is an automatic monitoring and early warning system which is low in cost, stable in operation and excellent in performance, the dependence of intelligent monitoring on a communication network is remarkably reduced, disaster monitoring and early warning can be completely moved forward to a field monitoring position original address, field multi-point deployment and control can be achieved, and systematic monitoring and early warning can be enhanced.

Description

Technical Field

[0001] The present invention relates to monitoring technology based on big data and machine learning, and in particular to a debris flow video recognition monitoring model based on video image recognition, and an automatic debris flow monitoring system adapted to edge devices, belonging to the technical fields of data recognition, electronic digital data processing, and geological disaster monitoring and early warning computing models and systems. Background Art

[0002] The intelligent debris flow early warning system automatically collects and calculates debris flow disaster environmental signals in real time, outputs disaster precursors and process analysis results, and provides timely data information support for scientific decision-making such as disaster warning, disaster process monitoring, and disaster emergency response. In this type of system, the video signal collected through video surveillance is an important original disaster information. As a signal type that records geological disaster environmental information, surveillance video has many advantages such as rich and intuitive information, clear spatial information, strong multi-target tracking capabilities, and support for image recognition and understanding. However, it also faces the disadvantages of huge data volume, high analysis cost, strong computing power requirements, and high equipment construction cost.

[0003] Since video signal analysis has high requirements for both analysis technology and equipment performance, in traditional debris flow monitoring systems, the video signals collected by the video monitoring terminal need to be transmitted to the system center / core computing unit for calculation and analysis. Since the video monitoring terminal for debris flow prediction is usually arranged in the upstream formation area of ​​the channel, and the center / core computing unit is usually arranged in the downstream exit area of ​​the channel with relatively better economic and traffic conditions, the early warning system needs to bear huge video signal transmission costs. With the rise and development of edge computing technology, existing technologies have developed solutions for debris flow video recognition and early warning using edge computing.

[0004] The existing technology "Fluid Disaster Detection and Warning Platform Based on Attention Mechanism Stream Network" (CN 117935487 A) provides a debris flow video stream monitoring and identification method, which uses a combination of deep learning algorithms and traditional opencv algorithms to detect the boundaries of fluid disasters such as debris flows and flash floods in video streams, and then compares them with the warning boundary threshold line to determine whether to trigger a disaster warning. Among them, the algorithm includes two steps, algorithm one is an attention-based stream network algorithm, and algorithm two is an overall algorithm for video stream detection. Algorithm one uses the attention mechanism to calculate the correlation between the features of two frames of images, improve the dynamic changes between the two frames of images, input the images of two consecutive frames into the network, and finally output the motion changes on the two consecutive frames of images through the algorithm. On this basis, algorithm two uses jump calculation to filter background factors such as non-flash floods to improve the detection effect of flash floods and other fluids.

[0005] The above-mentioned prior art has two defects: First, the analysis and processing of debris flow videos continues the traditional means of extracting image features or image flattening features and inputting the algorithm process to output the analysis results, and the amount of calculation is still huge. It is precisely because its algorithm 1 adopts this technical concept that the flow network with a simple attention mechanism detects any significant changes in the two frames of images, which seriously interferes with the detection of flash flood fluids. Therefore, it is necessary to superimpose algorithm 2 to eliminate the redundant features extracted by algorithm 1 for the detection and identification of flash flood fluid targets. Second, although edge computing devices are introduced into the debris flow monitoring system, the overall technical concept of the early warning scheme does not show adjustments that are compatible with edge computing. In other words, the extraction of flattening features, repeated iterative calculations around image features, elimination of redundant features, etc. in the algorithm all require large computing overheads; its technical solution can be realized on edge devices, mainly relying on the current high-performance edge computing devices, rather than based on the improvement of computing logic in video recognition and the reduction of computing power requirements brought about by algorithm refinement, and the cost of the technical solution is relatively high. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and provide a new logic for debris flow monitoring video analysis and calculation, so as to efficiently construct a debris flow video recognition model sample set; and further provide a method for constructing a debris flow video recognition model, and to build a debris flow intelligent monitoring system on a low-cost edge computing device.

[0007] To achieve the above object, the present invention first provides the following sample set construction method for debris flow video recognition model.

[0008] A sample set construction method for debris flow video recognition model. Get surveillance video V And video related parameters; Monitor video V Classified as positive sample videos containing debris flow activities PV Negative sample videos that do not contain debris flow activity content NV ; Construct positive sample set[ PS ],include: Step S100, read the positive sample video PV , mark the first frame as the background frame Fr 0 ; Step S200, set a cache frame queue of size n QC , load queue QC ,queue QC The first frame is recorded as Fr 1 , the last frame is recorded as Fr e , Step S300: Read the positive sample video frame by frame PV To current frame Fr i , Step S310, first current frame Fr i,i=1 Analyze and proceed to step S330; Step S320, from the second current frame Fr i,i>1 From the analysis, execute: Step S321, determine the previous frame Fr i-1 Is it a debris flow frame? PPF If yes, go to step S322, if no, go to step S330, the debris flow frame PPF is a frame image containing debris flow targets; Step S322, respectively calculate the current frame Fr i With end frame Fr e Histogram of , calculate the difference percentage between the two histograms s ;like s >Threshold S, cache current frame Fr i To Queue QC End frame Fr e , go to step S330, if s ≤ threshold S, update Fr i for Fr i+1 And proceed to step S320; Step S330 includes: Step S331, calculate the current frame Fr i With background frame Fr 0 The frame difference image is binarized to obtain the current long frame difference binary image LBI i ′, calculate the current long frame difference binary image LBI i The white proportion of ′ w i ,calculate w i and w i-1 The difference Δw i , w i-1 The last calculation w i The result value of like Δw i >0, the current long frame difference binary image LBI i 'Save as a long frame difference binary image LBI i , enter S332, like Δw i ≤0, discard the current long frame difference binary image LBI i ′ and update the current frame Fr i , count the number of discards k, If k<threshold K, go to step S331, If k ≥ threshold K, the current background frame Fr 0 Update to the current first frame Fr 1 , proceed to step S320; Step S332, calculate the current frame Fr i With the first frame Fr 1 The frame difference image is binarized to obtain the current short frame difference binary image SBI i , proceed to step S350; Step S340 includes: Step S341: convert the long frame difference binary image LBI i Perform an open operation and save the result graph OLBI i , the short frame difference binary image SBI i Perform the opening operation and display the result graph OSBI i Store in binary image queue QB , count queue QB Size m, if m < threshold M, repeat the short frame difference binary image SBI i Perform the opening operation and display the result graph OSBI i Store in queue QB Until m=threshold M, go to step S342; Step S342, read queue QB , the result graph OSBI i White pixels are mapped to the result image through M masks OLBI i Generate a multi-value grayscale image MGI i And save it to the positive sample set [ PS ],renewFr i And proceed to step S320; for any j Secondary Mask Mapping , j = 1, 2, 3, ... M, implement: Queue QB 1st~ j Results OSBI i Mask mapping to result map OLBI i And set the number of gray values ​​to j +2, of which the first is 255, the j +2 are 0, and the rest are based on the gradient d Arithmetic decrease; Process all positive sample videos PV , get the positive sample set[ PS ]; Process all negative sample videos NV , according to and construct the positive sample set [ PS ] The same method is used to construct the negative sample set [NS ]; Positive sample set[ PS ] and negative sample set[ NS ]The sample set that constitutes the debris flow video recognition model[ S ].

[0009] The basic concept of the sample set construction method of the debris flow video recognition model is to use grayscale images to represent the frame image information of the monitoring video. The key is to scientifically quantify the debris flow targets, various dynamic and static elements in complex environments, debris flow movement characteristics, etc. in the monitoring video with different grayscale values, and obtain sample images with rich grayscale value information, so that the information required for debris flow video recognition can be limited and fully expressed, which becomes the basis for refined machine learning operations.

[0010] The sample set construction method of the debris flow video recognition model can be optimized in the following aspects: Each optimization scheme can be implemented separately or simultaneously without conflict.

[0011] Optimization 1. Queue QC The size n is determined according to formula 1. Among them, FPS max —Maximum frame rate of the camera (in fps), D —The distance between the camera and the monitored target (in meters).

[0012] Formula 1 Optimization 2: Threshold K value camera maximum frame rate FPS max .

[0013] Optimization 3: Threshold M and Gradient d Satisfies the relationship in formula 2.

[0014] Formula 2.

[0015] Optimization 4: Separately divide the positive sample set [ PS ] and negative sample set[ NS ] All multi-value grayscale images MGI After image enhancement, the sample set is formed [ S ].

[0016] Utilizing the above-mentioned sample set construction method of the debris flow video recognition model, the present invention also provides the following debris flow video recognition model construction method.

[0017] A method for constructing a debris flow video recognition model, using the above debris flow video recognition model sample set construction method to construct a sample set [ S ], using machine learning to train a debris flow video recognition model.

[0018] In order to adapt to the performance of edge computing devices, the above-mentioned debris flow video recognition model construction method can use the Tiny Darknet model as the basis for machine learning and improve it.

[0019] The present invention also provides the following debris flow edge monitoring system.

[0020] A debris flow edge monitoring system, the hardware system uses the edge device as the core device, and the core device is connected to the debris flow monitoring camera; the software system includes a video acquisition module, a sample set construction module, a model training module, and a result output module; the video acquisition module obtains the debris flow monitoring video from the debris flow monitoring camera V The sample set construction module executes the above debris flow video recognition model sample set construction method to construct the sample set [ S ], the model training module installs the above debris flow video recognition model, and the result output module outputs the debris flow recognition result.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention discloses a technical concept of using debris flow monitoring video for analysis and identification to realize dynamic monitoring of debris flow, and the ultimate technical problem to be solved is to provide a low-cost edge monitoring solution applicable to field debris flow channel sites. To solve the ultimate technical problem, the technical concept of the present invention designs two technical improvement routes. One is to design the ultimate solution as an edge computing system, that is, to select edge computing devices with certain computing power but low cost as the core components of the system, and to ensure the system computing efficiency and stable operation, so that it can be promoted and applied in field debris flow channel sites with relatively backward economic, transportation and infrastructure conditions. Second, to support the aforementioned improvement route, the prior art concept of directly extracting and analyzing video data is abandoned, and the problem of video frame image feature analysis is converted into the problem of grayscale feature analysis of grayscale images, and the 256 grayscale values ​​in the grayscale image and their corresponding image meanings are fully utilized to reduce the huge and complex information of the debris flow and its environment captured by the monitoring video to a sufficient and limited range, thereby greatly reducing the computing overhead and adapting to the low-cost system resources with edge computing devices as the core. (2) To provide a final solution, the method for constructing a sample set of a debris flow video recognition model of the present invention provides a method for extracting image sample data from a debris flow monitoring video. Each image sample is a multi-value grayscale image containing dynamic information of the debris flow within a time window, and different grayscale values ​​in the image represent the debris flow morphology and environmental factors at different times. As a result, the sample data used for machine learning training takes into account the three goals of containing disaster dynamic information, removing interference from harsh environments, and simplifying the expression form of image features, and can simultaneously meet the requirements of achieving the recognition model training standard and the model running on edge computing devices. This technical concept has never been discussed in existing debris flow automatic monitoring and early warning technologies. (3) Based on the results of preliminary research, the method for constructing a debris flow video recognition model of the present invention provides an improved Tiny Darknet model as a machine learning training model. The improvement of the TinyDarknet model is based on two characteristics: in theory, the demand for grayscale calculation of sample data; in actual debris flow prevention and control projects, the derivative information of the viewing angle of the monitoring video lens in graphic recognition. In this way, the number of channels and the number of convolutional layers are reduced to balance the calculation amount and model performance requirements. (4) The debris flow edge monitoring system of the present invention is a low-cost, stable and high-performance automatic monitoring and early warning system for debris flow disasters. By placing all image analysis and calculations of the debris flow videos collected by the monitoring terminal on the edge computing device, the dependence of intelligent monitoring on the communication network is significantly reduced, so that the debris flow disaster monitoring and early warning can be "moved forward" to the field ditch site, effectively avoiding the situation when the disaster occurs that the central computing unit is restricted due to the interference of the harsh environment with the data transmission, resulting in the large-scale automatic monitoring and early warning being delayed, disconnected or even ineffective.Furthermore, since the investment and operation costs of the debris flow edge monitoring system of the present invention are low, it can be deployed and installed in more debris flow potential areas, or installed in a targeted manner in high-risk areas of debris flow channels, thereby achieving multi-point control of the entire river basin and enhancing the systematic intelligent monitoring and early warning capabilities of disasters. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It uses positive sample videos PV Construct positive sample set[ PS ] Schematic diagram of the main steps of the process.

[0023] Figure 2 Is a positive sample video PV 100 Frame image during an analysis.

[0024] Figure 3 Is a positive sample video PV 100 Binary plot from an analysis.

[0025] Figure 4 Is a positive sample video PV 100 A synthetic multi-value grayscale image from an analysis MGI i .

[0026] Figure 5 It is the ROC curve and PR curve of the classifier.

[0027] Figure 6 It's surveillance video V test Frame image.

[0028] Figure 7 It's surveillance video V test Grayscale image during recognition.

[0029] Figure 8 It is a schematic diagram of the software system framework of the debris flow edge monitoring system.

[0030] Fig. 9 This is a simplified diagram of the field layout of the debris flow edge monitoring system.

[0031] The numbers in the accompanying drawings are: 100 video acquisition module; 200 sample set construction module; 300 model training module; 400 result output module; 500 data transmission module. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present invention are further described below in conjunction with the accompanying drawings. Embodiment 1

[0033] like Figure 1 to Figure 4 As shown, the method of the present invention is used to construct a debris flow video recognition model sample set.

[0034] 1. Obtain debris flow monitoring video V Obtain debris flow monitoring video V , surveillance video V Including positive sample videos PV With negative sample video NV . Positive sample video PV Content records debris flow activity, negative sample video NV The content did not record any debris flow activity.

[0035] Positive sample video PV There are 21 videos in total, with video lengths ranging from 19s to 137s. The monitoring scenes include debris flow flume simulation experiments and debris flow channel field sites. NV There are 18 videos in total, with video lengths ranging from 23s to 109s. The monitoring scenes are field sites of debris flow channels and events or phenomena that interfere with debris flow identification occur, such as rain, water flowing / rising in the channel, and wind shaking the surrounding plants.

[0036] At the same time, obtain the relevant parameters of each video, including the maximum frame rate of the camera FPS max (Unit: fps), distance between camera and monitoring target D (Unit: m).

[0037] 2. Using positive sample videos PV Construct positive sample set[ PS ] Figure 1 It uses positive sample videos PV Construct positive sample set[ PS ] Schematic diagram of the main steps of the process.

[0038] Step S100, read the positive sample video PV , mark the first frame as the background frame Fr 0 .

[0039] First mark background frame Fr 0 When, by default Fr 0 Non-debris flow frame NPF , that is, the frame image that does not contain the debris flow target.

[0040] Step S200, set a cache frame queue of size n QC , load queue QC ,queue QC The first frame is recorded as Fr 1 , the last frame is recorded as Fr e .

[0041] In this embodiment, the value of n is determined according to Formula 1. PV 100 Take the analysis as an example: FPS max =24fps, D =376 m, n=24 / 2=12.

[0042] Step S300: Read the positive sample video frame by frame PV To current frame Fr i , analyze the current frame: Step S310, first current frame Fr i,i=1 Analyze and proceed to step S330; Fr i,i=1 During analysis, the first mark is already defaulted. Fr 0 for NPF , so just go directly to step S330.

[0043] Step S320, from the second current frame Fr i,i>1 To analyze the current frame, execute: Step S321, determine the previous frame Fr i-1 Is it a debris flow frame? PPF (Desert Frame PPF refers to a frame image containing a debris flow target), if yes, proceed to step S322, if no, proceed to step S330; Step S322, queue QC Similarity analysis, including: calculating the current frame separately Fr i With end frame Fr e Histogram of , calculate the difference percentage between the two histograms s , make conditional judgment: like s >Threshold S, cache current frame Fr i To Queue QC End frame Fr e , go to step S330, if s ≤ threshold S, update Fr i forFr i+1 And proceed to step S320.

[0044] In this implementation, the threshold S is set to 1%.

[0045] Figure 2 Is a positive sample video PV 100 Frame images from a certain analysis, where (a) background frame Fr 0 , (b) Current frame Fr i .

[0046] Step S330, queue QC Frame difference processing, including: Step S331, performing long frame difference processing, including: Calculate the current frame Fr i With background frame Fr 0 The frame difference image is binarized to obtain the current long frame difference binary image LBI i ′, calculate the current long frame difference binary image LBI i The white proportion of ′ w i ,calculate w i and w i-1 ( w i-1 Refers to the last calculation w i The result value of w i ,but w i-1 =0) Δw i , make conditional judgment; like Δw i >0, the current long frame difference binary image LBI i 'Save as a long frame difference binary image LBI i , enter S332, like Δw i ≤0, discard the current long frame difference binary image LBI i ′ and update the current frame Fr i (i.e. read the next frame to the current frame), count the number of discards k, and make conditional judgment: If k<threshold K, go to step S331, If k ≥ threshold K, the current background frame Fr 0 Cache as the current first frame Fr 1 (i.e. update queue QC ), go to step S320.

[0047] In this example, the threshold K is FPS max =24.

[0048] Step S332, performing short frame difference processing, including: Calculate the current frame Fr i With the first frame Fr 1 The frame difference image is binarized to obtain the current short frame difference binary image SBI i , enter step S340.

[0049] Figure 3 Is a positive sample video PV 100 Binary images in a certain analysis, where (a) long frame difference binary image LBI i , (b) Short frame difference binary image SBI i .

[0050] Step S340, synthesizing a multi-value grayscale image, includes: Step S341, performing an opening operation, including: converting the long frame difference binary image LBI i Perform an open operation and save the result graph OLBI i , the short frame difference binary image SBI i Perform the opening operation and display the result graph OSBI i Store in binary image queue QB , count queue QB Size m, if m < threshold M, repeat the short frame difference binary image SBI i Perform the opening operation and display the result graph OSBI i Store in queue QB Until m=threshold M, go to step S342.

[0051] In this implementation, the long frame difference binary image LBI i Short frame difference binary image SBI iThe 3×3 rectangular convolution kernels are used to perform opening operations to remove the tiny noise in the image. At the same time, the M value is calculated according to formula 2: d =10, M=24.

[0052] Step S342, performing image synthesis, including: reading the queue QB , the result graph OSBI i White pixels are mapped to the result image through M masks OLBI i Generate a multi-value grayscale image MGI i And save it to the positive sample set [ PS ],renew Fr i And proceed to step S320.

[0053] In the mask mapping, for any j Second-rate( j=1,2,3,…M ) Mask mapping, execution: Queue QB 1st~ j Results OSBI i Mask mapping to result map OLBI i And set the number of grayscale values ​​of the mask mapping result image to j +2, of which the first is 255, the j +2 are 0, and the rest are based on the gradient d Arithmetically decreasing, that is, the grayscale value of the result image is [255, (255- d ),(255-2 d ), …, (255 -jd ), 0].

[0054] In this embodiment, j = The result of 1 mask mapping has 3 grayscale values: [255, 245, 0]. j = The result of 2 mask mappings has 4 grayscale values: [255, 245, 235, 0]. j =M= The result of 24 mask mappings has 26 grayscale values: [255, 245, 235, ..., 15, 0]. After 24 mask mappings, a multi-value grayscale image with 26 grayscale values ​​is generated. MGI i , save to the positive sample set [ PS ].

[0055] Figure 4 Is a positive sample video PV 100 A synthetic multi-value grayscale image from an analysis MGI i .

[0056] Process all positive sample videos PV , get the positive sample set [ PS ].

[0057] In this implementation, 21 positive sample videos PV The frame images were extracted and processed at 10 frames / s, and a total of 738 multi-value grayscale images were obtained. MGI , forming the positive sample set [ PS ].

[0058] 3. Using negative sample videos NV Constructing negative sample set[ NS ] According to and construct the positive sample set[ PS ] The same method is used to process negative sample videos one by one NV , get the negative sample set [ NS ].

[0059] In this implementation, 18 negative sample videos NV The frame images were extracted and processed at 10 frames / s, and a total of 692 multi-value grayscale images were obtained. MGI , forming a negative sample set[ NS ].

[0060] 4. Obtain sample set S ] Merge positive sample set[ PS ] and negative sample set[ NS ], breaking up the image order and forming a sample set for the debris flow video recognition model [ S ]. Embodiment 2

[0061] The method of the present invention is used to construct a sample set of debris flow video recognition model. The same as the first embodiment is not repeated here. The difference is that in order to expand the sample set, the multi-value grayscale image is MGI Perform preprocessing.

[0062] The preprocessing specifically adopts image enhancement processing: the 21 positive sample videos in Example 1 are PV 738 multi-value grayscale images obtained MGI With 18 negative sample videos NV 692 multi-value grayscale images obtained MGI , and perform random rotation (rotation angle 0°~5°), zoom in, zoom out, and add image noise (addition probability 5%) respectively.

[0063] 4428 positive sample images were obtained to form the positive sample set [ PS]; 4152 negative sample images were obtained to form the negative sample set [ NS ]. Combined to form a sample set[ S ], containing 8580 multi-value grayscale images MGI . Embodiment 3

[0064] like Figure 5 to Figure 7 As shown, the sample set constructed using Example 2 [ S ] The method of the present invention is further used to construct a debris flow video recognition model.

[0065] 1. Divide into training set, test set, and validation set The sample set[ S ]The data are randomly divided into training set, test set and validation set in the ratio of 7:2:1.

[0066] 2. Training a binary classifier The machine learning model is based on the Tiny Darknet model (reference: Shi Lulu et al. Journal of Nanjing University of Posts and Telecommunications (Natural Science Edition), 2018, 38(4): 89-95. DOI: 10.14132 / j.cnki.1673-5439.2018.04.013) and improved to effectively reduce the amount of data transmission and data computing and storage overhead.

[0067] Improvements to the Tiny Darknet model include: ① Reducing the number of channels: Since the sample set images are grayscale images with a color channel of 1, the number of image channels in the original model is reduced to 1, which significantly reduces the amount of computation in the convolutional layer of the model, especially when the image size remains unchanged, reducing the computation by about 2 / 3, greatly improving the computational efficiency. ② Reducing convolutional layers: Since the debris flow monitoring perspective has been specially designed, image target detection and positioning are not required in image analysis, so the relevant convolutional layers for performing detection tasks are deleted; and since the grayscale image samples have few redundant features, the convolutional layers can be appropriately reduced. After comprehensive testing, it is most appropriate to reduce the 16 convolutional layers of the original model to 8.

[0068] Using the sample set [ S ] Train a binary classifier.

[0069] 3. Classifier effect test The confusion matrix obtained through training is shown in Table 1. In the table, TP is the number of samples correctly classified in the positive class, TN is the number of samples correctly classified in the negative class, FP is the number of samples misclassified as positive in the negative class, and FN is the number of samples misclassified as negative in the positive class. The number of samples is N = 8580. The classifier has a good balance between positive and negative sample classification (as shown in Table 2).

[0070] Table 1 Confusion matrix of training results Predict positive class Predict negative class Actual positive category TP=3823 FN=482 Actual negative class FP=452 TN=3700 Table 2 Classifier performance index Overall accuracy Ac Model accuracy Pr Recall F1 score Specific Sp Score (TP+TN)) / N=87.6% TP / (TP+FP)=89.4% TP / (TP+FN)=88.9% 2Pr*Re / (Pr+Re)=89.0% TN / (TN+FP)=89.1% The ROC and PR curves showed that the AUC values ​​of the classifiers were both approximately 0.9, indicating that the classifiers had good discrimination capabilities. Figure 5 It is the ROC curve and PR curve of the classifier.

[0071] 4. Example verification Using an embodiment 1 debris flow monitoring video V Other debris flow monitoring videos V test Test the classifier on examples.

[0072] When the 11th frame of the image was read and analyzed, the probability of the image being identified as a debris flow was 87.3%. The classifier performed well.

[0073] Figure 6 It's surveillance video V test Frame images, where (a) frame 1 Fr 1 , (b) Frame 5 Fr 5 , (c) Frame 11 Fr 11 ; Figure 7 It's surveillance video V test Grayscale image in recognition, where (a) long frame difference binary image LBI , (b) Short frame difference binary image SBI , (c) Multi-value grayscale image MGI .

[0074] The trained classifier model is the constructed debris flow video recognition model. Embodiment 4

[0075] like Figure 8-Figure 9 As shown, the debris flow video recognition model constructed in Example 3 is used to further develop a debris flow edge monitoring system.

[0076] The debris flow edge monitoring system includes a hardware system and a software system.

[0077] The hardware system uses the edge device as the core device, which is connected to the debris flow monitoring camera and also includes necessary input and output peripherals, power supply equipment, and communication lines. In this implementation, the edge device uses a Raspberry Pi 4B, equipped with a 1.5G processor, 4G memory, a USB interface connected to the debris flow monitoring camera (1280*720 pixels), a 5V 3A interface connected to the solar power supply equipment, and a Gigabit Ethernet card connected to the 4G communication module communication line (such as SIM7600CE).

[0078] Figure 8 It is a schematic diagram of the software system framework of the debris flow edge monitoring system.

[0079] The software system includes a video acquisition module 100, a sample set construction module 200, a model training module 300, a result output module 400, and a necessary data transmission module 500. The video acquisition module 100 uses the data transmission module 500 to obtain the debris flow monitoring video from the debris flow monitoring camera. V The sample set construction module 200 executes the sample set construction method of the debris flow video recognition model of the present invention to construct the sample set [ S ], the model training module 300 runs the debris flow video recognition model constructed in Example 3, and the result output module 400 outputs the debris flow recognition result.

[0080] The software system is written into the edge device, specifically: on the operating system of the Raspberry Pi 4B, the C++ version of OpenCV is compiled and installed to realize the functions of obtaining real-time frames of the camera and performing image preprocessing. Subsequently, the debris flow video recognition model developed in Example 3 is compiled and installed on the Raspberry Pi 4B.

[0081] In order to test the compatibility of the edge device in the monitoring system and the debris flow video recognition model (i.e., the model developed in Example 3), a classification operation compatibility comparison is set (Table 3). In the process from reading samples to completing classification, the video image recognition rate of the method of the present invention on the Raspberry Pi 4B can fully reach the maximum frame rate of the camera 24fps, that is, it can perform intelligent recognition of the captured video without delay.

[0082] Table 3 Comparative test of classification operation suitability Darknet method Method of the present invention Edge Devices Raspberry Pi 4B Raspberry Pi 4B Image Sample Directly extract video frame images Multi-value grayscale image MGI The average maximum frame rate from reading samples to completing classification 1.6 frames / s 4.5 frames / s Classification accuracy 69% 88% The above debris flow edge monitoring system was installed in Jiangjiagou, Dongchuan District, Yunnan Province on June 3, 2024 for real-time monitoring of debris flows in the field. Fig. 9 This is a simplified diagram of the field layout of the debris flow edge monitoring system. Fig. 9In the figure, the camera icon in the upper picture indicates the installation location of the debris flow ditch of the edge monitoring system, and the full set of edge monitoring systems is deployed on the installation column in the lower picture, that is, from camera shooting to output recognition results, all are completed at the original site of the camera icon in the upper picture. The system has been running stably for 6 months, and the recognition results of real-time monitoring video are consistent with the actual records of multiple observations.

Claims

1. A method for constructing a sample set of a debris flow video recognition model, characterized in that: Get surveillance video V And video related parameters; Monitor video V Classified as positive sample videos containing debris flow activities PV Negative sample videos that do not contain debris flow activity content NV ; Construct positive sample set[ PS ], include: Step S100, read the positive sample video PV , mark the first frame as the background frame Fr 0; Step S200, set a cache frame queue of size n QC , load queue QC ,queue QC The first frame is recorded as Fr 1. The last frame is recorded as Fr e , Step S300: Read the positive sample video frame by frame PV To current frame Fr i , Step S310, first current frame Fr i,i=1 Analyze and proceed to step S330; Step S320, from the second current frame Fr i,i>1 From the analysis, execute: Step S321, determine the previous frame Fr i-1 Is it a debris flow frame? PPF If yes, go to step S322, if no, go to step S330, the debris flow frame PPF is a frame image containing debris flow targets; Step S322, respectively calculate the current frame Fr i With end frame Fr e Histogram of , calculate the difference percentage between the two histograms s ;like s >Threshold S, cache current frame Fr i To Queue QC End frame Fr e , go to step S330, if s ≤ threshold S, update Fr i for Fr i+1 And proceed to step S320; Step S330 includes: Step S331, calculate the current frame Fr i With background frame Fr 0 frame difference image and perform binarization processing to obtain the current long frame difference binary image LBI i ′, calculate the current long frame difference binary image LBI i The white proportion of ′ w i ,calculate w i and w i-1 The difference Δw i , w i-1 The last calculation w i The result value of like Δw i >0, the current long frame difference binary image LBI i 'Save as a long frame difference binary image LBI i , enter S332, like Δw i ≤0, discard the current long frame difference binary image LBI i ′ and update the current frame Fr i , count the number of discards k, If k<threshold K, go to step S331, If k ≥ threshold K, the current background frame Fr 0 is updated to the current first frame Fr 1, proceed to step S320; Step S332, calculate the current frame Fr i With the first frame Fr 1 frame difference image and perform binarization processing to obtain the current short frame difference binary image SBI i , proceed to step S350; Step S340 includes: Step S341: convert the long frame difference binary image LBI i Perform an open operation and save the result graph OLBI i , the short frame difference binary image SBI i Perform the opening operation and display the result graph OSBI i Store in binary image queue QB , count queue QB Size m, if m < threshold M, repeat the short frame difference binary image SBI i Perform the opening operation and display the result graph OSBI i Store in queue QB Until m=threshold M, go to step S342; Step S342, read queue QB , the result graph OSBI i White pixels are mapped to the result image through M masks OLBI i Generate a multi-value grayscale image MGI i And save it to the positive sample set [ PS ],renew Fr i And proceed to step S320; for any j Secondary Mask Mapping , j = 1, 2, 3, ... M, implement: Queue QB 1st~ j Results OSBI i Mask mapping to result map OLBI i And set the number of gray values ​​to j +2, of which the first is 255, the j +2 are 0, and the rest are based on the gradient d Arithmetic decrease; Process all positive sample videos PV , get the positive sample set[ PS ]; Process all negative sample videos NV , according to and construct the positive sample set [ PS ]The same method is used to construct negative sample sets[ NS ]; Positive sample set[ PS ] and negative sample set[ NS ]The sample set that constitutes the debris flow video recognition model[ S ].

2. The method for constructing a debris flow video recognition model sample set according to claim 1, characterized in that: The queue QC The size n is determined according to formula 1, Formula 1 In the formula, FPS max —Maximum frame rate of the camera, in fps, D —The distance between the camera and the monitoring target, in meters.

3. The method for constructing a debris flow video recognition model sample set according to claim 1, characterized in that: The threshold K is the maximum frame rate of the camera. FPS max .

4. The method for constructing a debris flow video recognition model sample set according to claim 1, characterized in that: The threshold M and the gradient d Satisfying formula 2, Formula 2.

5. The method for constructing a debris flow video recognition model sample set according to claim 1, characterized in that: The positive sample set [ PS ] and the negative sample set [ NS ] All multi-value grayscale images MGI After image enhancement, the sample set is formed [ S ].

6. The method for constructing a debris flow video recognition model sample set according to claim 5, characterized in that: The image enhancement includes: MGI Randomly rotate 0°~5°, zoom in, zoom out, and add 5% image noise.

7. A method for constructing a debris flow video recognition model, characterized in that: The sample set is constructed by using the method for constructing a debris flow video recognition model sample set according to any one of claims 1 to 6. S ], using machine learning to train a debris flow video recognition model.

8. The method for constructing a debris flow video recognition model according to claim 7, characterized in that: The machine learning adopts an improved Tiny Darknet model, and the improvements of the Tiny Darknet model include: setting the number of image channels to 1 and setting the number of convolutional layers to 8.

9. Debris flow edge monitoring system, characterized by: The hardware system uses an edge device as a core device, and the core device is connected to a debris flow monitoring camera; the software system includes a video acquisition module (100), a sample set construction module (200), a model training module (300), and a result output module (400); the video acquisition module (100) obtains debris flow monitoring video from the debris flow monitoring camera V The sample set construction module (200) executes the debris flow video recognition model sample set construction method according to any one of claims 1 to 6 to construct a sample set [ S ], the model training module (300) installs the debris flow video recognition model constructed according to claim 7 or 8, and the result output module (400) outputs the debris flow recognition result.

10. The debris flow edge monitoring system according to claim 9, characterized in that: The edge device is a Raspberry Pi, on which the C++ version of OpenCV and the debris flow video recognition model are compiled and installed.

Images