A real-time recognition method for drainage status of river outlets based on video images

The drainage state model of river outlets established through deep learning technology uses motion vector analysis of video images to solve the real-time and accuracy of river outlet flow water recognition in the existing technology, and realizes automatic identification and pollution of river outlet flow water Monitoring improves the identification accuracy and monitoring efficiency of drainage status.

CN113343923BActive Publication Date: 2024-12-03JIANGSU YUTU INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110746229.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-01
Publication Date
2024-12-03
Estimated Expiration
2041-07-01

AI Technical Summary

Technical Problem

The existing river drainage outlet flow recognition technology is not yet mature, and it is difficult to detect abnormal drainage in real time relying on manual monitoring. There are errors and noise interference in video monitoring, making it difficult to effectively identify drainage status and control sewage discharge secretly.

Method used

Deep learning technology is adopted to establish a deep learning model for drainage state of the river channel based on video images. Through motion vector analysis of video data and combined with convolutional neural network training, real-time identification and abnormal alarm of the flow state of the river channel outlet is achieved.

Benefits of technology

Real-time unmanned monitoring and automatic identification of the flow state of the river outlet is realized, which improves the identification accuracy and efficiency, can promptly discover problems and eliminate pollution risks, and enhances the robustness of drainage state recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113343923B_ABST
    Figure CN113343923B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for real-time recognition of drainage status of a river outlet based on video images, comprising the following steps: first, establishing a deep learning model of drainage status, and the steps of establishing the deep learning model of drainage status are: collecting videos of drainage and non-drainage status of the river outlet; splitting the video into frames of images; calculating the motion vector of each frame of image; marking the motion vector with drainage status label information; establishing a deep learning model of drainage status; and training the deep learning model of drainage status. Then, a real-time video stream of the river outlet is obtained through a camera, the video stream is split into frames of images, the motion vector of each frame of image is calculated, and the motion vector of each frame of image is analyzed and output with the established deep learning model of drainage status to output a preliminary drainage status result, analyze and filter the preliminary results, and output the final drainage status information. The present invention realizes real-time unmanned monitoring and automatic recognition of drainage status of the river outlet, which is beneficial for relevant departments to timely discover problems, eliminate hidden dangers, and solve the problem of river pollution from the source.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of smart water services, and in particular to a real-time recognition method for the drainage status of a river outlet based on video images, which is suitable for real-time video recognition of water flow at a river outlet. Background Art

[0002] With the development of society, especially the progress of science and technology, the rapid development of social productivity has been greatly promoted; in particular, the widespread application of communication network technology has enabled most fields to begin to develop from traditional manual statistical analysis to intelligent direction.

[0003] The technology for identifying water flow at river outlets is not mature yet. It can be realized through video surveillance technology. If there is abnormal discharge at the outlet, it can be easily retrieved and viewed on the spot. However, the disadvantage of video surveillance is that it requires human observation to detect abnormalities. Therefore, it is imperative to develop an automatic identification method for water flow at river outlets to improve the accuracy and efficiency of intelligent identification of water flow at river outlets.

[0004] In recent years, deep learning has made important and successful breakthroughs in the field of artificial intelligence, becoming a new and popular research direction in machine learning. It has powerful learning and efficient feature expression capabilities, and has achieved great success in many fields such as computer vision, image and video analysis, speech recognition, and multimedia. Among them, the automatic recognition method of river outlet flow based on video images has high recognition accuracy and efficient video stream data analysis efficiency.

[0005] In the prior art, the drainage situation of river outlets is complicated. For example, the shapes of the outlets are different, which can be circular, square, polygonal, etc.; the water flow at the outlet varies greatly, and sometimes the water flow is very small, and there is a large error in measuring it through a liquid level meter; the color of the discharged water varies greatly, and sometimes it is mixed with the moss at the outlet. Therefore, it is an urgent problem to provide a method that can measure the drainage status of the outlet and then control the illegal discharge of sewage from the river outlet, especially the small amount of toxic sewage. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method for analyzing river outlet water flow video data with deep learning technology as the core, adopts deep convolutional neural network for training, adds multiple scene data to the data set, and obtains a judgment model of the outlet water flow scene after training. Through quasi-real-time analysis of the video stream, a real-time recognition method for the river outlet drainage status based on video images is realized, and abnormal drainage alarm information with time series properties is given in real time.

[0007] The object of the present invention is achieved in that:

[0008] A method for real-time recognition of drainage status of a river outlet based on video images, firstly establishing a deep learning model of drainage status, the steps of establishing the deep learning model of drainage status are: collecting videos of drainage and non-drainage status of the river outlet; splitting the videos into frames of images; calculating the motion vector of each frame of image; annotating the motion vector with drainage status label information; establishing a deep learning model of drainage status; training the deep learning model of drainage status; then obtaining a real-time video stream of the river outlet through a camera, splitting the video stream into frames of images, calculating the motion vector of each frame of image, using the established deep learning model of drainage status to analyze the motion vector of each frame of image to output a preliminary drainage status result, analyzing and filtering the preliminary result, and outputting the final drainage status information;

[0009] The method of establishing the model of river outlet drainage and setting the initial conditions is as follows:

[0010] Establishing a river outlet drainage model

[0011] u t +uu x =0..................(1)

[0012] To describe the drainage status of the outlet;

[0013] t represents time, x represents the position of the water surface, and u represents the velocity wave; x=0 represents the state position where the water will discharge from the pipe but has not discharged from the pipe. When x≤0, all the water is in the pipe, and its speed is much slower than that after it leaves the pipe. Therefore, it is assumed that u=0. After the water leaves the pipe, its speed suddenly increases, and the water flows smoothly through the drop. Assume that its average speed is v 0 , so the initial condition of equation (1) can be defined as

[0014]

[0015] Assume that δ is the minimum drainage velocity, so when the velocity v 0 When >δ>0, the drainage can be determined and subsequent quantitative analysis including flow can be performed;

[0016] The solution of equation (1) is in the region {x>0,t>x}, and the velocity wave is Explain that the outlet velocity changes with time; its change can be described by motion vector, and then the outlet drainage state can be obtained by finding the motion vector; the motion vector is the difference between two frames of images stored in graphic compression, which is a vector that describes the change of the spatial position of an object;

[0017] The calculation method of the motion vector v of the image is as follows: drain the adjacent two frames of grayscale image (g i , g j) is used to define the grayscale image of the motion vector as

[0018] g v =g i -g j ,

[0019] The motion vector v is the one-dimensional expansion of its grayscale image; the motion vector can be further improved into several adjacent motion vectors v i The weighted average value v = ∑a i v i ; or use the OptionalFlowFarneback algorithm to obtain v i , and use v = ∑a i v i The motion vector is obtained by weighted average; the above weighted average can also be arithmetic average or Gaussian weighted average;

[0020] The specific steps of the method for establishing the drainage state deep learning model are as follows:

[0021] (1) Collect a large number of drainage and non-drainage videos;

[0022] (2) Calculate the motion vector of each frame in the video to form a motion vector grayscale map; the motion vector grayscale map can be approximated by the difference between the previous and next two frames or determined by the OptionalFlowFarneback algorithm;

[0023] (3) Labeling the drainage state information for each motion vector grayscale image, where the label value is the state of the water flow corresponding to the motion vector. The specific labeling method is as follows: the label information types are divided into no water flow state, low-speed water flow state, medium-speed water flow state, and high-speed water flow state. The specific flow velocity values ​​of the low-speed water flow state, medium-speed water flow state, and high-speed water flow state are adjusted according to the specific situation, and generally 0 m / s, 1 m / s, 2 m / s, and 5 m / s can be used;

[0024] (4) Establishing a deep learning model of drainage status, which is composed of a number of convolutional graphs and a number of fully connected graphs. The convolutional layer and the fully connected layer of the model are set according to the situation.

[0025] (5) Training model: The conventional CNN convolutional neural network training method is used. The recognition accuracy reaches more than 99% when convergence, and the training learning rate is 0.0001.

[0026] Positive and beneficial effects: 1. The present invention realizes real-time unmanned monitoring and automatic identification of the drainage status of the river outlet, which is beneficial for relevant departments to discover problems in time, eliminate hidden dangers, and solve the river pollution problem from the source.

[0027] 2. To filter the preliminary results of the deep learning model for drainage status, this method filters and analyzes the preliminary results of the deep learning model for drainage status due to the complex drainage environment, poor drainage image quality, and large noise interference. In the case of unreasonable sudden changes in drainage volume, the method filters and processes them according to the situation at the time to improve the recognition rate of practical applications. For example, if one or several consecutive frames are judged as drainage but a large number of adjacent frame images are judged as no drainage, it is considered that the preliminary result is caused by image noise interference, and the final result does not report drainage.

[0028] 3. Use the drainage state deep learning model to judge the drainage state and the average velocity of the water flow at the outlet v 0 Affected by factors including lighting, environment, weather changes, brightness and day and night interference, it is difficult to extract it from the video. Because of the video image, the accuracy of directly judging the drainage status of the outlet using the motion vector calculated by the image is not high; the shape of the water flow at the outlet also causes a large difference in the motion vector grayscale image, so it is difficult to directly determine the water flow state through the motion vector grayscale image; this patent uses a deep learning method to establish a deep learning model of the drainage status, and uses the model to determine whether to drain, so as to improve the drainage status recognition accuracy and enhance its robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the steps of real-time identification of drainage status of a river outlet based on video images of the present invention;

[0030] Figure 2 A schematic diagram of the steps for establishing a deep learning model of drainage status of the present invention;

[0031] Figure 3 The discharge port image of the present invention;

[0032] Figure 4 It is a schematic diagram of the analytical solution of the outlet model of the present invention;

[0033] Figure 5 It is the grayscale image of the motion vector of the outlet image of the present invention;

[0034] Figure 6 Convert motion vector grayscale image to motion vector

[0035] Figure 7 Schematic diagram of the drainage status deep learning model architecture of the present invention. DETAILED DESCRIPTION

[0036] The following is a detailed description in conjunction with the accompanying drawings and specific implementations:

[0037] A method for real-time recognition of drainage status of a river outlet based on video images includes the following steps: first, establishing a deep learning model of drainage status, and the steps of establishing the deep learning model of drainage status are: collecting videos of drainage and non-drainage status of the river outlet; splitting the video into frames; calculating the motion vector of each frame; marking the motion vector with drainage status label information; establishing a deep learning model of drainage status; and training the deep learning model of drainage status. Then, a real-time video stream of the river outlet is obtained through a camera, the video stream is split into frames, the motion vector of each frame is calculated, and the established deep learning model of drainage status is used to analyze the motion vector of each frame to output a preliminary drainage status result, and the preliminary result is analyzed and filtered to output the final drainage status information.

[0038] Establish the river outlet drainage model and set the initial conditions as follows:

[0039] Establishing a river outlet drainage model

[0040] u t +uu x =0………………(1)

[0041] To describe the drainage status of the outlet, such as Figure 3 As shown;

[0042] In equation (1), t represents time, x represents the position of the water surface, and u represents the velocity wave.

[0043] x=0 means the state where the water will be discharged from the pipe but has not been discharged from the pipe. When x≤0, all the water is in the pipe, and its speed is much slower than that after it leaves the pipe. Therefore, it is assumed that u=0. After the water leaves the pipe, its speed suddenly increases, and the water flows smoothly through the drop. Assume that its average speed is v 0 , so the initial condition of equation (1) can be defined as

[0044]

[0045] Assume that δ is the minimum drainage velocity, so when the velocity v 0 When >δ>0, drainage can be determined and subsequent quantitative analysis, such as flow rate, can be performed.

[0046] The solution of equation (1) is Figure 4 As shown, in the region {x>0,t>x}, the velocity wave is It shows that the outlet velocity changes with time. Its change can be described by motion vector, and then the outlet drainage status can be obtained by finding the motion vector. The motion vector is the difference between two frames of images stored in graphic compression. It is a vector that describes the change of the spatial position of an object.

[0047] The motion vector v of the image is calculated as follows:

[0048] Use the drain port to drain the adjacent two frames of grayscale images (g i , g j ) to define the grayscale image of the motion vector (see Figure 5 )for

[0049] g v =g i -g j ,

[0050] The motion vector v is the one-dimensional expansion of its grayscale image (see Figure 6 ). The motion vector can be further improved into several adjacent motion vectors v i The weighted average value v = ∑a i v i Or use the OptionalFlowFarneback algorithm to obtain v i , and use v = ∑a i v i The motion vector is obtained by weighted average. The above weighted average can also be arithmetic average or Gaussian weighted average.

[0051] Use the drainage state deep learning model to judge the drainage state. The average velocity of the water flow at the outlet v 0 It is difficult to extract from the video because the video image is subject to a lot of noise interference, such as lighting, environment, weather changes, brightness, day and night, etc., so the accuracy of directly judging the drainage status of the outlet using the motion vector calculated by the image is not high; the shape of the water flow at the outlet also causes a large difference in the motion vector grayscale image, so it is difficult to directly determine the water flow status through the motion vector grayscale image. This patent uses a deep learning method to establish a deep learning model for drainage status, and uses the model to determine whether to drain, so as to improve the drainage status recognition accuracy and enhance its robustness.

[0052] The specific steps of the method for establishing a deep learning model of drainage status are as follows:

[0053] (1) Collect a large number of drainage and non-drainage videos;

[0054] (2) Calculate the motion vector of each frame in the video. For the specific calculation method, see point 3 of the specific implementation method to form a motion vector grayscale map. The motion vector grayscale map can be approximated by the difference between the previous and next two frames of images, or can be determined by a related algorithm such as OptionalFlowFarneback.

[0055] (3) Label the drainage status information for each motion vector grayscale image, where the label value is the state of the water flow corresponding to the motion vector. The specific labeling method is as follows: the label information types are divided into no water flow state, low-speed water flow state, medium-speed water flow state, and high-speed water flow state. The specific flow velocity values ​​of the low-speed water flow state, medium-speed water flow state, and high-speed water flow state are adjusted according to the specific situation, and generally 0 m / s, 1 m / s, 2 m / s, and 5 m / s can be used.

[0056] (4) Establishing a drainage state deep learning model, which is composed of a plurality of convolutional graphs and a plurality of fully connected graphs. The convolutional layer and the fully connected layer of the model can be set according to the situation, and this patent does not limit it. Figure 7 ), which is also the model we adopted.

[0057] (5) Training model: The conventional CNN convolutional neural network training method is used. The recognition accuracy reaches more than 99% when convergence, and the training learning rate is 0.0001.

[0058] The steps of the real-time recognition method of the drainage status of the river outlet based on video images are as follows:

[0059] (1) Obtain real-time video stream of the river outlet through the camera;

[0060] (2) Split the video stream into frames;

[0061] (3) Calculate the motion vector of each frame of image;

[0062] (4) Use the established drainage state deep learning model to analyze the motion vector of each frame image and output preliminary drainage state results;

[0063] (5) Analyze the preliminary drainage status results of the outlet given by the filtering model: Filter the preliminary results of the deep learning model for drainage status. Due to the complex environment of the outlet, poor drainage image quality, and large noise interference, this method filters and analyzes the preliminary results of the deep learning model for drainage status. In the case of unreasonable sudden changes in drainage volume, filtering is performed according to the situation at the time to improve the recognition rate of practical applications. For example, if one or several consecutive frames are judged to be drainage but a large number of adjacent frame images are judged to be non-drainage, it is considered that the preliminary result is caused by image noise interference, and the final result does not report drainage.

[0064] The present invention realizes the real-time unmanned monitoring and automatic identification of the drainage status of the river outlet, which is conducive to the relevant departments to find problems in time, eliminate hidden dangers, and solve the river pollution problem from the source; the drainage status is judged by the deep learning model of the drainage status, and the average velocity of the water flow at the outlet v 0Affected by factors including lighting, environment, weather changes, brightness and day and night interference, it is difficult to extract it from the video. Because of the video image, the accuracy of directly judging the drainage status of the outlet using the motion vector calculated by the image is not high; the shape of the water flow at the outlet also causes a large difference in the motion vector grayscale image, so it is difficult to directly determine the water flow state through the motion vector grayscale image; this patent uses a deep learning method to establish a deep learning model of the drainage status, and uses the model to determine whether to drain, so as to improve the drainage status recognition accuracy and enhance its robustness.

Claims

1. A method for real-time recognition of drainage status of a river outlet based on video images, characterized in that: First, a deep learning model of drainage status is established. The steps of establishing the deep learning model of drainage status are as follows: collect videos of drainage and non-drainage status of the river outlet; split the video into frames; calculate the motion vector of each frame; annotate the motion vector with drainage status label information; establish a deep learning model of drainage status; train the deep learning model of drainage status; then obtain the real-time video stream of the river outlet through the camera, split the video stream into frames, calculate the motion vector of each frame, use the established deep learning model of drainage status to analyze the motion vector of each frame, output the preliminary drainage status result, analyze and filter the preliminary result, and output the final drainage status information; The method of establishing the model of river outlet drainage and setting the initial conditions is as follows: Establishing a river outlet drainage model and t +uu x =0................(1) To describe the drainage status of the outlet; t represents time, x represents the position of the water surface, and u represents the velocity wave; x = 0 represents the state position where the water is about to discharge from the pipe but has not discharged from the pipe. When x ≤ 0, all the water is in the pipe, and its speed is much slower than that after it leaves the pipe. Therefore, it is assumed that u = 0. After the water leaves the pipe, its speed suddenly increases, and the water flows smoothly through the drop. Assuming that its average speed is v0, the initial condition of equation (1) can be defined as Assume that δ is the minimum drainage velocity, so when the velocity v0>δ>0, the drainage can be determined and subsequent quantitative analysis including flow can be performed; The solution of equation (1) is in the region {x>0,t>x}, and the velocity wave is It shows that the outlet velocity changes with time; its change can be described by motion vector, and then the outlet drainage state can be obtained by finding the motion vector; the motion vector motionvector is the difference between two frames of images stored in graphic compression, and it is a vector that describes the change of the spatial position of an object.

2. The method for real-time recognition of drainage status of a river outlet based on video images according to claim 1, characterized in that: The calculation method of the motion vector v of the image is as follows: drain the adjacent two frames of grayscale image (g i , g j ) is used to define the grayscale image of the motion vector as g v =g i -g j , The motion vector v is the one-dimensional expansion of its grayscale image; the motion vector is further improved into several adjacent motion vectors v i The weighted average value v = ∑a i v i ; or use the algorithm OptionalFlowFarneback method to obtain v i , and use v = ∑a i v i The motion vector is obtained by weighted averaging; the above weighted averaging adopts arithmetic averaging or Gaussian weighted averaging.

3. The method for real-time recognition of drainage status of a river outlet based on video images according to claim 1 is characterized in that: The specific steps of the method for establishing the drainage state deep learning model are as follows: (1) Collect a large number of drainage and non-drainage videos; (2) Calculate the motion vector of each frame in the video to form a motion vector grayscale map; the motion vector grayscale map can be approximated by the difference between the previous and next two frames of the image or determined by the OptionalFlowFarneback algorithm; (3) Labeling the drainage state information for each motion vector grayscale image, where the label value is the state of the water flow corresponding to the motion vector. The specific labeling method is as follows: the label information types are divided into no water flow state, low-speed water flow state, medium-speed water flow state, and high-speed water flow state. The specific flow velocity values ​​of no water flow state, low-speed water flow state, medium-speed water flow state, and high-speed water flow state are adjusted according to the specific situation, and specifically correspond to 0 m / s, 1 m / s, 2 m / s, and 5 m / s respectively; (4) Establishing a deep learning model of drainage status, which is composed of a number of convolutional graphs and a number of fully connected graphs. The convolutional layer and the fully connected layer of the model are set according to the situation. (5) Training model: The CNN convolutional neural network training method is used. The recognition accuracy reaches more than 99% when convergence, and the training learning rate is 0.0001.