A real-time detection system for water flow rate

Abstract

The present invention discloses a real-time detection system for water flow rate, which belongs to the field of water flow rate detection technology, including: a model conversion module for adapting the user's model configuration parameters to the water flow rate measurement model developed based on the edge device; a device-side WEB interface for data interaction with the edge device; a model reasoning and optimization module for optimizing the adapted water flow rate measurement model through a hybrid AI acceleration inference engine; a business processing module for pushing the reasoning results to the server, and an APP packaging and deployment module for expanding or replacing the original functions in the device; the present invention provides a computing power camera-based real-time monitoring of water flow rate, completes real-time detection of water flow rate at the edge, and then returns key information such as detection results and alarms to the central server. Compared with the traditional central-end computing mode, the present invention reduces the bandwidth consumption of data transmission, reduces the detection cost, and improves the efficiency of water flow rate detection.

Description

Technical Field

[0001] The present invention relates to the technical field of water flow rate detection, and in particular to a real-time detection system for water flow rate. Background Art

[0002] Water velocity is one of the most basic data in hydrological research and the foundation of all hydrological research. Due to the complex flow characteristics of natural rivers and the harsh outdoor test environment, the measurement of river velocity has always been a difficult task in hydrological testing. Especially in the case of frequent extreme situations such as floods, the need to obtain key information such as river velocity in a timely manner is more urgent. However, the traditional central-end calculation method has the problem of high cost and low efficiency due to the large amount of data collected by the front-end camera. Uploading all the data to the central end for calculation will lead to high costs and low efficiency. Even if there are network problems or central node failures, it will cause service interruptions, resulting in significant losses. In terms of security, if all data is transmitted back to the main server, the data will be extremely vulnerable to attacks during the operation. If the data is stored and analyzed locally, the security impact can be greatly reduced. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a real-time detection system for water flow rate, comprising:

[0004] Model conversion module, used to adapt the user's model configuration parameters to the water flow velocity measurement model developed based on the edge device;

[0005] The device-side WEB interface interacts with the edge device to observe the edge device status and the operation and adaptation of the water flow velocity measurement model, and to control the edge device and the water flow velocity measurement model;

[0006] Model reasoning and optimization module, which is used to optimize the adapted water flow velocity measurement model through a hybrid AI acceleration inference engine to support user-configured nodes to run on different engines to meet maximum performance, best compatibility, and maximum memory utilization;

[0007] The business processing module is used to push the results of optimized model reasoning to the server;

[0008] The APP packaging and deployment module is used to deploy custom algorithm projects by supporting uploading and running third-party APPs to expand or replace the original functions in the device.

[0009] Preferably, the model conversion module is used to build a network according to the operators supported by the edge device, train the water flow velocity measurement model, and convert the trained model into an offline model that can be used on the edge device.

[0010] Preferably, the device-side WEB interface is used to design a Basic Config column on the web front end based on the Web / JS and React components provided by the edge device to configure whether to enable detection, the ROI area of ​​detection, the warning value, whether to warn, and whether to push the detection results; configure the server-side IP / PORT through the Net Config column; manage the log level through Log Level; and manage user permissions through user info.

[0011] Preferably, the model conversion module is also used to use the cross-compilation tool of the edge device to recompile the opencv source code to adapt it to the device environment, and deploy the compiled opencv library in the project.

[0012] Preferably, the model conversion module is also used to read the data of three sub-channels of the YUV image in the memory through the YUV module of the device SDK development tool, and synthesize them into a complete YUV image; use opencv to convert the YUV image into an RGB image, and resize it to the input size of the algorithm model; use the calcOpticalFlowPyrLK function to calculate the optical flow map of the RGB image as the input data of the algorithm model; and train the water flow velocity measurement model based on the input data.

[0013] Preferably, the model reasoning and optimization module is also used to obtain the algorithm detection status, the ROI to be detected and the warning value information from the web front-end configuration, perform reasoning operations based on the configuration information, and run the model nodes on different engines according to the hybrid AI acceleration reasoning engine. Multiple accelerators are used in combination to achieve fast reasoning of the model. The reasoning result is the water flow velocity result predicted by the model.

[0014] Preferably, the business processing module is also used to use the OSD module to render the flow rate results of the model detection to the preview interface of the device background, and then the server side actively pulls the RTSP video stream of the device, and synchronizes the detection results and the video screen to the server side in real time. When the model inference result of the current frame has a large jitter compared with the historical frame, the current frame result is saved but not rendered or pushed.

[0015] Preferably, the business processing module is also used to use the httplib component to listen to server-side requests and push flow rate results, use the cross-compilation tool chain of the edge device to compile the JSONCPP component adapted to the device environment, and deploy it to the project to parse and package the corresponding http results and push them.

[0016] Preferably, the business processing module is also used to cooperate with the corresponding interface opened by the server backend to provide real-time feedback and push of flow rate results and alarm status, wherein:

[0017] Send a request to the server through the / api / slave / edge / [alg_uuid] / camera / [ip] / token interface to obtain the task_uuid of the device's current algorithm task, and push the flow rate results and alarms to the server through the / api / slave / result interface.

[0018] Preferably, the APP packaging and deployment module is used to package the project code and install it to the device background, start the APP in the device background, and check the APP running status.

[0019] The present invention discloses the following technical effects:

[0020] The present invention provides a computing camera-based real-time monitoring of water flow rate, completes real-time detection of water flow rate at the edge, and then returns key information such as detection results and alarms to the central server. Compared with the traditional central computing mode, the present invention reduces the bandwidth consumption of data transmission, reduces the detection cost, and improves the efficiency of water flow rate detection.

[0021] The hybrid AI accelerated inference engine provided by the present invention completes the efficient operation of offline models on devices, supports running network nodes on different engines, and uses multiple accelerators in combination, thereby meeting the requirements of maximum performance, best compatibility, and maximum memory utilization.

[0022] In the system design provided by the present invention, the algorithm project of the present invention is used with the device-side WEB interface, and the algorithm operation status and related parameters can be controlled from the web interface outside the algorithm by logging into the device background. And this part of the function can be conveniently expanded according to the needs.

[0023] The present invention has an open technical module expansion capability. In the present invention, the technical modules can be easily expanded or replaced, which is suitable for diversified algorithm and function development, so that the algorithm development is no longer completely limited to the technical modules provided by the device.

[0024] The present invention has open interactive capabilities. The present invention introduces a local web end to expand the interactive capabilities of users and algorithm ends, introduces an HTTP module to complete the remote interactive capabilities of the edge end and the server end, and develops an http interface that can access requests and responses from other system ends. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a schematic diagram of the system architecture of the present invention. DETAILED DESCRIPTION

[0027] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0028] like Figure 1 As shown, the present invention aims at the defects in the above-mentioned prior art and the advantages of the unified architecture, and the technical problems to be solved by the present invention are embodied in the following points:

[0029] Development of a water flow velocity measurement model on edge devices. The traditional optical flow calculation algorithm is replaced by a deep learning optical flow model, and the network nodes of the model need to be within the computing power support range of the device.

[0030] AI acceleration engine. It integrates high-performance computing engines to optimize neural networks and provides hybrid reasoning basic solutions from tools to engines, ensuring that algorithm models can be efficiently reasoned on devices with limited computing resources.

[0031] Develop various perception capabilities of hardware devices. Integrate various perception capabilities of hardware devices to complete efficient and stable edge transplantation of algorithms. Complete encoding and decoding of video image data, obtain device system parameters, support display of intelligent results and linkage alarms based on hardware devices.

[0032] Real-time feedback and push of results. The edge detection results are fed back to the server backend in real time.

[0033] The present invention is based on intelligent front-end computing equipment and designs a water flow velocity measurement algorithm to complete a real-time water flow velocity detection system. The present invention is jointly completed by the following modules:

[0034] Model conversion: The computing camera provides a set of hybrid reasoning basic solutions from tools to engines. The model conversion tool is provided to convert the model into an offline model that can run on the device based on the user's model configuration parameters. Since the computing power of the computing camera, as well as the supported operators, network layers, and network nodes are limited, when performing model conversion, it is necessary to modify the model configuration according to the device's support to adapt it to the device's model conversion capabilities.

[0035] Device-side WEB interface design: The computing camera supports custom whole-machine Web. The present invention designs the camera-side private water flow velocity measurement algorithm Web interface based on the JS_sdk provided by the device. Interact with the water flow velocity measurement algorithm end in the Web interface to realize the control of the device and algorithm.

[0036] Model pre- and post-processing and reasoning: The computing power camera provides a hybrid AI acceleration reasoning engine, which is responsible for the efficient operation of offline models on the device. It supports running user-configured nodes on different engines to meet maximum performance, best compatibility, and maximum memory utilization. The present invention processes network models by manually configuring and autonomously selecting different acceleration units such as ARM, DSP, FPGA, GPU, etc., making full use of the relative advantages of different acceleration units in processing different types of data, in order to achieve a comprehensive balance in memory, bandwidth, and performance.

[0037] Business processing module: The results generated by model reasoning need to be pushed to the server. The present invention divides this business into two parts. The first part is to render the speed measurement results generated by model reasoning on the device front end, and the server actively pulls the rendering results from the device end; the second part is to send real-time alarms and push the flow rate values ​​generated by model reasoning to the server end.

[0038] APP packaging and deployment: The device supports uploading and running third-party APPs to deploy custom algorithm projects to expand or replace the original functions of the device. Use the cross-compilation tool chain and APP packaging tool provided by the device to compile and package the custom water flow speed measurement algorithm project code, and deploy, start, and view it through the device background.

[0039] Based on the hardware development kit provided by the device, the present invention performs custom deployment of necessary components. It mainly uses technical modules such as YUN, SYSTEM, LOG, OSD, etc. provided by the device SDK development tool, as well as other technologies including deep learning algorithm model conversion and reasoning, front-end technologies such as WEB / JS / REACT, RTSP video push and pull streaming, HTTP remote connection, etc.

[0040] The overall design and implementation steps of the system proposed by the present invention are as follows:

[0041] Step 1: Model conversion. Use operators supported by the computing camera to build a network, train the water flow velocity measurement network model, and use the model conversion tool to convert the model into an offline model that can be used on the device.

[0042] Step 2: Design the device-side WEB front-end interface. The web interface is an interface for real-time interaction with users after the algorithm program is installed and deployed. The present invention is based on the development components such as Web / JS and React provided by the device. The BasicConfig column is designed on the web front end to configure whether to enable detection, the ROI area of ​​detection, the warning value, whether to warn, whether to push the detection results and other configuration items. The Net Config column is designed to configure the IP / PORT of the server. In addition, the Log Level is designed to manage the log level, and the user info is designed to manage user permissions.

[0043] Step 3: Compile the opencv component. Since the input of the algorithm model used in the present invention is the optical flow field of the image, it is necessary to first use the calcOpticalFlowPyrLK function in opencv to obtain the optical flow map of the image, but the device does not have the ability of the opencv framework. The present invention uses the cross-compilation tool of the device to recompile the opencv source code to adapt it to the device environment, and deploys the compiled opencv library in the project.

[0044] Step 4: Image preprocessing. The smart front-end computing camera will encode and save the real-time water flow video in YUV format. The present invention uses the YUV module of the device SDK development tool to read the data of the three sub-channels of the YUV image in the memory and synthesize it into a complete YUV image. Use opencv to convert the YUV image into an RGB image and resize it to the input size of the algorithm model. Use the calcOpticalFlowPyrLK function to calculate the optical flow map of the RGB image as the input data of the algorithm model.

[0045] Step 5: Model reasoning. Obtain information such as the algorithm detection status, ROI to be detected, and warning value from the web front-end configuration, and perform reasoning operations based on the configuration information. If the current detection status is configured as true, perform reasoning. Based on the hybrid AI acceleration reasoning engine provided by the device, the model node is run on different engines such as ARM, DSP, and CUDA. Multiple accelerators are used in combination to achieve fast reasoning of the model. The reasoning result is the water flow velocity result predicted by the model.

[0046] Step 6: Algorithm post-processing. The model inference results will have some inevitable jitters and outliers due to the influence of external factors. The present invention screens the model inference results. If the current model inference results have a large jitter compared with the historical frames, the current frame results are saved but not rendered or pushed.

[0047] Step 7: Rendering of business results. Use the OSD module provided by the device to render the flow rate results of the model detection to the preview interface of the device background. If the detection in step 6 is abnormal, render the previous result. The server side then actively pulls the RTSP video stream of the device and synchronizes the detection results and video screen to the server side in real time.

[0048] Step 8: Business component configuration. The present invention uses the httplib component to monitor server-side requests and push flow rate results. Since the requests and response results interacting with the server need to be encapsulated in JSON format, and the device does not have a JSON component, the present invention uses the device's cross-compilation tool chain to compile the JSONCPP component that adapts to the device environment, and deploys it to the project to parse and package the corresponding http results. httplib is a general open source component and does not need to be recompiled.

[0049] Step 9: Pushing business results. Based on the configuration of the corresponding httplib and jsoncpp components in step 8, the present invention cooperates with the corresponding interface opened by the server backend to perform real-time feedback and push of flow rate results and alarm status. Send a request to the server through the / api / slave / edge / [alg_uuid] / camera / [ip] / token interface to obtain the task_uuid of the current algorithm task of the device, and push the flow rate results and alarms to the server through the / api / slave / result interface.

[0050] Step 10: APP packaging and deployment. Steps 1-9 completed the local construction of the water flow velocity measurement algorithm project code. In this step, the packaging tool provided by the device is used to package the project code and install it to the device background. The APP is started in the device background to check the APP running status.

[0051] Through the above steps, real-time monitoring and flow rate detection of water flow can be finally achieved based on the smart front-end camera.

[0052] The present invention deploys the water flow detection algorithm on the camera side, saving bandwidth and computing resources. The use of hybrid AI accelerated reasoning can ensure real-time detection of water flow rate when the device side has only low computing power, and then return the detection results to the video background. It reduces resource consumption and ensures real-time detection.

[0053] The present invention estimates water flow velocity based on image time-frequency analysis. It is necessary to set different matching templates for different river sections, and the algorithm has many preconditions. This patent detects water flow velocity based on a machine vision deep learning model, which has high detection accuracy and reusability. And it corrects and screens possible abnormal problems in the detection. This patent migrates the water flow velocity measurement model to the computing power edge device, and uses a hybrid acceleration inference engine for efficient inference, which can ensure the real-time and accuracy of the algorithm.

[0054] The present invention uses a smart front-end computing camera and its supporting hardware open platform as a carrier, and completes water flow rate detection based on a machine vision deep learning algorithm. It involves edge computing, model transplantation, and AI accelerated reasoning, and realizes the construction of a smart front-end water flow rate detection system for the water conservancy industry.

[0055] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0057] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A real-time detection system for water flow rate, characterized in that: include: Model conversion module, used to adapt the user's model configuration parameters to the water flow velocity measurement model developed based on the edge device; The device-side WEB interface performs data interaction with the edge device, is used to observe the edge device status and the operation and adaptation of the water flow velocity measurement model, and to control the edge device and the water flow velocity measurement model; Model reasoning and optimization module, which is used to optimize the adapted water flow velocity measurement model through a hybrid AI acceleration inference engine to support user-configured nodes to meet maximum performance, best compatibility, and maximum memory utilization when running on different engines; The business processing module is used to push the results of optimized model reasoning to the server; APP packaging and deployment module, which is used to deploy custom algorithm projects by supporting uploading and running third-party APPs to expand or replace the original functions in the device; The model conversion module is also used to read the data of three sub-channels of the YUV image in the memory through the YUV module of the device SDK development tool, and synthesize them into a complete YUV image; use opencv to convert the YUV image into an RGB image, and resize it to the input size of the algorithm model; use the calcOpticalFlowPyrLK function to calculate the optical flow map of the RGB image as the input data of the algorithm model; and train the water flow velocity measurement model based on the input data; The model reasoning and optimization module is also used to obtain the algorithm detection status, the ROI to be detected and the warning value information from the web front-end configuration, perform reasoning operations according to the configuration information, and run the model nodes on different engines according to the hybrid AI acceleration reasoning engine. Multiple accelerators are used in combination to achieve fast reasoning of the model. The reasoning result is the water flow velocity result predicted by the model. The business processing module is also used to use the OSD module to render the flow rate results of the model detection to the preview interface of the device background, and then the server side actively pulls the RTSP video stream of the device, and synchronizes the detection results and the video screen to the server side in real time. Among them, when the model inference result of the current frame is greatly jittered compared with the historical frame, the current frame result is saved but not rendered or pushed.

2. A real-time detection system for water flow rate according to claim 1, characterized in that: The model conversion module is used to build a network according to the operators supported by the edge device, train the water flow velocity measurement model, and convert the trained model into an offline model that can be used on the edge device.

3. A real-time detection system for water flow rate according to claim 2, characterized in that: The device-side WEB interface is used to design a Basic Config column on the web front end based on the Web / JS and React components provided by the edge device to configure whether to enable detection, the ROI area of ​​detection, the warning value, whether to issue a warning, and whether to push the detection result configuration items; configure the IP / PORT of the server side through the Net Config column; Manage the log level through Log Level; manage user permissions through user info.

4. A real-time detection system for water flow rate according to claim 2, characterized in that: The model conversion module is also used to use the cross-compilation tool of the edge device to recompile the opencv source code to adapt it to the device environment, and deploy the compiled opencv library in the project.

5. A real-time detection system for water flow rate according to claim 1, characterized in that: The business processing module is also used to use the httplib component to listen to server-side requests and push flow rate results, use the cross-compilation tool chain of the edge device to compile the JSONCPP component that adapts to the device environment, and deploy it to the project to parse and package the corresponding http results and push them.

6. A real-time detection system for water flow rate according to claim 5, characterized in that: The business processing module is also used to cooperate with the corresponding interface opened by the server backend to provide real-time feedback and push of flow rate results and alarm status, wherein: Send a request to the server through the / api / slave / edge / [alg_uuid] / camera / [ip] / token interface to obtain the task_uuid of the device's current algorithm task, and push the flow rate results and alarms to the server through the / api / slave / result interface.

7. A real-time detection system for water flow rate according to claim 6, characterized in that: The APP packaging and deployment module is used to package the project code and install it to the device background, start the APP in the device background, and check the APP running status.

Images