A man-machine cooperative processing method for urban ditches and rivers based on video recognition

Abstract

The application discloses a kind of urban inner ditch river man-machine collaborative processing method based on video recognition, it is related to video recognition or understanding field, the data acquisition and analysis closed loop covering whole cycle before, middle and after heavy rain is constructed: before rainfall, using unmanned aerial vehicle and camera collect before and after management image, generate the management operation influence atlas of quantitative evaluation management effect;During rainfall, use fixed camera to continuously monitor water level and flow rate;After rainfall, image is collected again and abnormal river section is identified again.By taking camera data as boundary condition, combined with management influence atlas to drive hydrological model to carry out inversion, the missing heavy rainfall process data is deduced, and finally the accurate attribution of abnormal cause is realized and the differentiated management scheme is generated.The application effectively solves the problem of missing monitoring data under adverse weather, realizes the quantitative evaluation of management effect and the scientific diagnosis of river abnormal cause.

Description

Technical Field

[0001] This invention relates to the field of general control or regulation systems, specifically to a human-machine collaborative processing method for urban ditches and rivers based on video recognition. Background Technology

[0002] Urban canals and rivers are a crucial component of urban drainage systems, and their unobstructed flow directly impacts the effectiveness of urban flood control. Traditional management methods primarily rely on periodic manual inspections and reactive emergency dredging, which suffer from drawbacks such as low efficiency, numerous blind spots, and the inability to quantitatively assess management effectiveness. In recent years, although drones and cameras have been introduced for monitoring, these methods are largely limited to data collection and have failed to form an effective closed loop with management actions.

[0003] In particular, a key challenge in disaster analysis lies in the fact that during periods of heavy rainfall, severe weather conditions render mobile monitoring equipment such as drones inoperable, resulting in the lack of hydrological data (such as flow rate and water level) collected at key river sections throughout the entire rainfall process. This data gap makes it difficult to accurately determine whether flooding is caused by incomplete management, insufficient design capacity, or sudden blockage during post-disaster post-disaster analysis, thus affecting the targetedness and scientific nature of subsequent management decisions. Summary of the Invention

[0004] The purpose of this invention is to provide a human-machine collaborative processing method for urban ditches and rivers based on video recognition, which aims to overcome the above-mentioned problems existing in the prior art.

[0005] To achieve the objective, the present invention provides the following technical solution:

[0006] A human-machine collaborative processing method for urban inner river based on video recognition includes the following steps: Step S1: In the first preset time before heavy rainfall, use cameras pre-deployed at key nodes and drones flying along flight paths to collect a first inner river image dataset before treatment; perform manual treatment operations and record treatment operation information; use cameras and drones to collect a second inner river image dataset after treatment.

[0007] Step S2: Based on the above-mentioned governance operation information set, compare and analyze the above-mentioned first and second inner canal image datasets, quantitatively analyze the river section range affected by the governance operation and the quantitative indicators of the improvement of the river channel flow capacity, and generate a governance operation impact map.

[0008] Step S3: During the heavy rainfall, key point image data are continuously collected using cameras, and the data of the first heavy rainfall process is estimated in real time using video image analysis technology; the data of the first heavy rainfall process includes water level and surface flow velocity.

[0009] Step S4: Within a second preset time period after the heavy rainfall ends, collect a third inland river image dataset using cameras and drones.

[0010] Step S5: By comparing and analyzing the above-mentioned second and third inner canal image datasets, identify abnormal river sections that have experienced water accumulation, siltation, or structural damage.

[0011] Step S6: For each of the above-mentioned abnormal river segments, perform the following sub-steps:

[0012] Step S61: Using the first heavy rainfall process data of the key nodes associated with the abnormal river section as the boundary condition, the second inner ditch river image dataset as the initial state, and the third inner ditch river image dataset as the final state, call the hydrological and hydraulic model to perform reverse inference and calibration, and deduce the spatiotemporal distribution data of water level and flow in the abnormal river section during the heavy rainfall process, as the second heavy rainfall process data.

[0013] Step S62: Combining the second inner canal image dataset, the second heavy rainfall event data, the third inner canal image dataset, and the impact map of the governance operation, comprehensively analyze the causes of the anomaly and generate a subsequent governance plan that includes the specific location, governance measures, and priorities.

[0014] Furthermore, the aforementioned first, second, and third inland river image datasets all include at least image data acquired by drones, water spectral images, and key point image data acquired by cameras.

[0015] Furthermore, in step S1 above, the treatment operation information set includes the location, type, and quantitative content of the treatment operation; the quantitative content includes the volume of dredging, the length of the culvert to be cleared, and the volume and type of obstacles to be removed.

[0016] Furthermore, the aforementioned impact map of the governance operation is a visual data layer generated based on a geographic information system, which records and correlates the spatial location of the governance operation and its quantitative improvement indicators on the river's flow capacity.

[0017] Furthermore, in step S2, the aforementioned quantitative indicators characterize the rate of change in the cross-sectional area of ​​the river channel before and after treatment.

[0018] Further, in step S1, the GIS coordinates of all key nodes with deployed cameras are uploaded to construct a GIS topology network; in step S61, the data of the first heavy rainfall process of the key nodes associated with the abnormal river section are used as boundary conditions, specifically: based on the GIS topology network, the two key nodes with deployed cameras located upstream and downstream of the abnormal river section are selected; if multiple tributaries flow into the GIS topology network, the key nodes of the main tributary upstream of the confluence point are selected; the water level and surface velocity estimated by these selected key nodes in step S3 are used as boundary conditions.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] Firstly, this invention constructs a closed-loop data acquisition and analysis system covering the entire cycle of heavy rainfall, effectively solving the problem of missing key hydrological data under severe weather conditions. Specifically, it establishes a second image dataset of the treated inner canals by using drones for detailed scanning of the entire area before heavy rainfall; during heavy rainfall, it continuously collects process data such as water level and surface flow velocity using fixed-deployment cameras; and after heavy rainfall, it conducts another full-area scan to obtain a third image dataset of the inner canals. This three-stage data architecture, especially by utilizing cameras to overcome the limitation of drones being unable to operate in severe weather, completely records the entire process of heavy rainfall, providing an irreplaceable data foundation for accurate post-disaster reassessment.

[0021] Secondly, this invention enables quantitative evaluation of the effectiveness of governance measures and precise attribution of the causes of river anomalies, promoting the upgrade of governance models from experience-driven to data-driven. Specifically, this invention innovatively introduces the intermediate product of governance operation impact maps. By comparing high-precision three-dimensional data before and after governance, the actual effect of each dredging, clearing, and other governance operations can be quantitatively evaluated. In post-disaster analysis, not only are the pre- and post-disaster states compared, but the above quantitative evaluation results are also used as key input parameters for correction of hydrological and hydraulic models. Combined with the inverted heavy rainfall process data, the causes of river anomalies can be scientifically distinguished, thereby generating targeted and differentiated post-disaster governance plans, greatly improving the scientific nature of governance decisions and the efficiency of fund utilization. Attached Figure Description

[0022] Figure 1 This is a flowchart of the human-machine collaborative processing method for urban ditches and rivers based on video recognition in this invention.

[0023] Figure 2 This is a structural block diagram of the human-machine collaborative system for urban ditches and rivers based on video recognition in this invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below with reference to specific embodiments. These embodiments are implemented based on the technical solutions of this invention, providing detailed implementation methods and specific operating procedures; however, the scope of protection of this invention is not limited to the following embodiments.

[0025] like Figure 1 and Figure 2 As shown, implementing the method of this invention requires constructing a human-machine collaborative system that combines hardware and software. The hardware includes multiple cameras, at least one drone, at least one smart mobile terminal, and a data fusion and analysis platform.

[0026] The cameras are high-definition smart cameras with built-in video image analysis modules, which can estimate water level and surface flow velocity in real time through video image analysis technology. These cameras are deployed at key nodes of the inner canals. Among them, the key nodes of the inner canals mainly include: (1) the confluence of main streams and tributaries; (2) the inlet and outlet of culverts and box culverts; (3) the upstream and downstream of control structures such as sluice gates and overflow dams; (4) bends or narrowings where the river geometry changes abruptly; (5) sections that are historically prone to siltation and blockage; and (6) the outlets that eventually discharge into external water bodies.

[0027] The drones are used for large-scale patrols and are equipped with high-resolution visible light cameras and multispectral sensors to acquire image data and water spectral images of inland rivers.

[0028] The smart mobile terminal is used by inspection personnel to report treatment operation information.

[0029] The data fusion and analysis platform is deployed in the command center. On the software side, the platform integrates a Geographic Information System (GIS), a time-series database, a computer vision analysis module, and hydrological and hydraulic models for river flow simulation and inversion. The hydrological and hydraulic models can receive river topology and cross-sectional morphology defined by the GIS as initial conditions, and use water level and flow velocity estimated from camera image data as boundary conditions to perform one-dimensional or two-dimensional hydrodynamic simulations, outputting spatiotemporal distribution data of water level and flow. Such models can be implemented using mature commercial or open-source software in the field, such as locally calibrated SWMM models or Mike Urban models.

[0030] It should be noted that, in the method of this invention, "heavy rainfall" refers to a clearly defined heavy rainfall event when the meteorological department issues a rainstorm warning signal or higher (such as a yellow, orange, or red rainstorm warning) for the city. Of course, a heavy rainfall event can also be determined based on a quantitative rainfall threshold, such as an expected rainfall intensity ≥ 20 mm / hour, or an expected 24-hour cumulative rainfall ≥ 50 mm.

[0031] In this invention, the aforementioned heavy rainfall event is divided into three stages: before occurrence, during occurrence, and after occurrence. The period before heavy rainfall refers to 24 to 72 hours prior to the predicted start time of heavy rainfall according to weather forecasts. The period during heavy rainfall refers to the time from the actual start of rainfall until the rainfall essentially stops and river levels begin to recede significantly. The period after heavy rainfall refers to the 24 hours following the cessation of rainfall, when surface runoff has largely drained into rivers and river levels have returned to normal or controllable levels.

[0032] A human-machine collaborative processing method for urban canals and rivers based on video recognition includes the following steps:

[0033] Step S1: Within the first preset time before the occurrence of heavy rainfall (this first preset time can be adjusted according to the amount of governance tasks, usually within the range of 24-72 hours, such as 48 hours before the expected occurrence of heavy rainfall), execute the following sub-steps:

[0034] Step S11: Use cameras pre-deployed at key nodes and drones flying along flight paths to collect the first inland river image dataset before treatment.

[0035] Specifically, after the meteorological department issues a heavy rainfall warning, operators activate the contingency plan 48 hours before the expected rainfall. In step S11, the data fusion analysis platform automatically generates a drone patrol route covering all main streams and tributaries of the inland rivers. The drone flies along the route, collecting image data and water spectral images covering all main streams and tributaries of the inland rivers. The data fusion analysis platform automatically generates a full-area realistic 3D model, along with its corresponding high-precision 3D dense point cloud data and digital elevation model, through photogrammetric processing. Simultaneously, all cameras are activated to acquire image data of key nodes. All data is aggregated into the first inland river image dataset.

[0036] Preferably, in step S11, the GIS coordinates of all key nodes are uploaded to the data fusion analysis platform to construct a GIS topology network.

[0037] Step S12: The inner canal is manually treated and the treatment operation information is recorded.

[0038] Specifically, patrol personnel conduct remediation operations on suspected blockage points in the inner canals and report the remediation operation information in the form of remediation work orders via smart mobile terminals (such as mobile apps), which are then compiled into a remediation operation information set. The remediation operation information includes at least: location (such as GIS coordinates), type (such as dredging / clearing / obstacle removal / repair), and quantitative content (such as dredging volume, culvert length, volume and type of obstruction removed, etc.).

[0039] As a preferred option, the data fusion analysis platform generates reports of suspected blockage points based on the first inland river image dataset for use by patrol personnel. Of course, patrol personnel can also identify suspected blockage points through on-site inspections or by viewing images uploaded by drones and cameras.

[0040] Step S13: After the treatment operation is completed, use cameras and drones again to collect image datasets of the treated second inner canal.

[0041] Specifically, after the treatment operation is completed, the same full-domain data collection as step S11 is performed again to obtain the above-mentioned second inner canal image dataset. That is, the second inner canal image dataset includes at least image data acquired by UAV through multi-view aerial photography, used to generate a real-world 3D model of the river channel, a high-precision 3D dense point cloud and a digital elevation model, water spectral images, and key point image data acquired by cameras.

[0042] Step S2: Based on the aforementioned governance operation information set, the data fusion analysis platform compares and analyzes the first and second inland river image datasets, quantitatively analyzes and generates quantitative indicators representing the river section affected by the governance operations and the improvement of the river's flow capacity, forming a governance operation impact map. These quantitative indicators include, but are not limited to, the rate of change in the river's cross-sectional area before and after governance.

[0043] Specifically, the data fusion analysis platform automatically associates the governance operation information set from step S12 with the first inner ditch river image dataset from step S11 and the second inner ditch river image dataset from step S13. For example, for a governance work order for the XX Road culvert, the data fusion analysis platform extracts the river cross-section from the identical locations in the 3D point cloud data before and after governance. Through comparative analysis, it automatically calculates the area change rate of the cross-sectional area after governance (e.g., the cross-sectional area increased by 15%), which is the quantitative result of the improvement in the river's flow capacity. The data fusion analysis platform binds the quantitative result to the governance work order and visualizes it on a visualization data layer (e.g., a GIS map) generated based on a Geographic Information System (GIS) using different colors and range layers, forming a governance operation impact map. This governance operation impact map records and associates the spatial location of the governance operation and its quantitative improvement indicators of the river's flow capacity.

[0044] Step S3: During the heavy rainfall, key point image data are continuously collected using cameras, and the data of the first heavy rainfall process is estimated in real time using video image analysis technology; the data of the first heavy rainfall process includes water level and surface flow velocity.

[0045] Specifically, after the heavy rainfall began, drones were grounded due to safety regulations. All cameras continuously recorded video, acquiring key point image data, which was then uploaded to the data fusion and analysis platform. The computer vision analysis module of the data fusion and analysis platform analyzed the key point image data in real time using video image analysis technology and estimated the data for the first heavy rainfall event. For example, the computer vision analysis module identified water level gauges pre-placed on the bank in the key point image data and recorded the water level time-series curve; in addition, the computer vision analysis module tracked the movement of natural floating objects (such as leaves and foam) in the key point image data, selected two virtual baselines with known distances, calculated the time difference of the floating objects passing through, and thus estimated the surface flow velocity.

[0046] Step S4: Within a second preset time after the heavy rainfall ends (e.g., within 24 hours after the rainfall ends), use the aforementioned cameras and drones to collect a third inland river image dataset.

[0047] Step S5: The data fusion analysis platform identifies abnormal river sections experiencing water accumulation, siltation, or structural damage by comparing and analyzing the second and third inland river image datasets mentioned above. Step S6: For each of the aforementioned abnormal river sections, the data fusion analysis platform performs the following sub-steps:

[0048] Step S61: Using the first heavy rainfall process data of the key nodes associated with the abnormal river section as the boundary condition, the second inner ditch river image dataset as the initial state, and the third inner ditch river image dataset as the final state, call the hydrological and hydraulic model to perform reverse inference and calibration, and deduce the spatiotemporal distribution data of water level and flow in the abnormal river section during the heavy rainfall process, as the second heavy rainfall process data.

[0049] Specifically, in step S61, the selection strategy for the key nodes associated with the abnormal river section is as follows: the data fusion analysis platform selects the two key nodes with cameras deployed upstream and downstream of the abnormal river section based on the GIS topology network; if multiple tributaries flow into the GIS topology network, the key node of the main tributary upstream of the confluence point is selected.

[0050] The data fusion analysis platform uses the water level and surface velocity estimated in step S3 for these selected key nodes as boundary conditions, and the second inland river image dataset after the abnormal river section was treated in step S13 as the initial state. It then calls upon the platform's built-in hydrological and hydraulic model for back-analysis and calibration, aiming to ensure that the simulation results are consistent with the actual situation reflected in the third inland river image dataset. For example, through iterative calculations, the average error between the simulated water level at the main cross-section of the river and the water level markers identified in the images is made less than a preset threshold (e.g., 5 cm), and the contours of the water accumulation area are basically matched. Through this back-analysis and calibration, the spatiotemporal distribution data of water level and flow rate in this abnormal river section, which cannot be directly measured during rainfall, are calculated.

[0051] Step S62: Combining the second inner canal image dataset, the second heavy rainfall event data, the third inner canal image dataset, and the impact map of the governance operation, comprehensively analyze the causes of the anomaly and generate a subsequent governance plan that includes the specific location, governance measures, and priorities.

[0052] For example, regarding a specific abnormal river section, the data fusion analysis platform discovered: 1) The second inland river image dataset shows that the river channel was unobstructed after treatment; 2) The second heavy rainfall event data shows that the peak flow of the abnormal river section far exceeded the design standard; 3) The third inland river image dataset shows that the abnormal river section had large-scale water accumulation; 4) The impact map of the treatment operation confirmed that there were no unfinished treatment defects before the occurrence of this heavy rainfall. Based on this comprehensive analysis, the platform inferred that the main cause of the waterlogging in this abnormal river section was: rainfall intensity exceeding the design standard, insufficient drainage capacity of the XX river section, and non-maintenance issues. The generated subsequent treatment plan is: for the XX river section, the treatment priority is medium, the short-term measure is to clean up the residual garbage after the water recedes, and the long-term measure is to carry out engineering renovations to improve drainage capacity.

[0053] The embodiments described are for illustrative purposes only and are not intended to limit the scope of the invention. Any modifications, equivalent substitutions, or improvements made without departing from the spirit and scope of the invention should be included within the protection scope of the invention.

Claims

1. A human-machine collaborative processing method for urban ditches and rivers based on video recognition, characterized in that, Includes the following steps: Step S1: Within the first preset time before the heavy rainfall, use cameras pre-deployed at key nodes and drones flying along flight paths to collect the first inland river image dataset before the treatment; perform manual treatment operations and record the treatment operation information set; A dataset of images of the second inner canal after treatment was collected using cameras and drones; Step S2: Based on the governance operation information set, compare and analyze the first and second inland river image datasets, quantitatively analyze the river section range affected by the governance operation and the quantitative indicators of the improvement of the river channel flow capacity, and generate a governance operation impact map. Step S3: During the heavy rainfall, key point image data are continuously collected using a camera, and the data of the first heavy rainfall process is estimated in real time using video image analysis technology; the data of the first heavy rainfall process includes water level and surface flow velocity; Step S4: Within a second preset time period after the heavy rainfall ends, collect a third inland river image dataset using cameras and drones; Step S5: By comparing and analyzing the second and third inland river image datasets, identify abnormal river sections that have experienced water accumulation, siltation, or structural damage; Step S6: For each of the aforementioned abnormal river segments, perform the following sub-steps: Step S61: Using the first heavy rainfall process data of the key nodes associated with the abnormal river section as the boundary condition, the second inner canal image dataset as the initial state, and the third inner canal image dataset as the final state, call the hydrological and hydraulic model to perform reverse inference and calibration, and deduce the spatiotemporal distribution data of water level and flow in the abnormal river section during the heavy rainfall process, as the second heavy rainfall process data. Step S62: Combining the second inner canal image dataset, the second heavy rainfall event data, the third inner canal image dataset, and the impact map of the governance operation, comprehensively analyze the causes of the anomaly and generate a subsequent governance plan that includes the specific location, governance measures, and priorities.

2. The method according to claim 1, characterized in that, The first, second, and third inland river image datasets each include at least image data acquired by drones, water spectral images, and key point image data acquired by cameras.

3. The method according to claim 1, characterized in that, In step S1, the treatment operation information set includes the location, type, and quantitative content of the treatment operation; the quantitative content includes the volume of dredging, the length of the culvert to be cleared, and the volume and type of obstacles to be removed.

4. The method according to claim 1, characterized in that, The impact map of the governance operation is a visual data layer generated based on a geographic information system. It records and associates the spatial location of the governance operation and its quantitative improvement indicators on the river's flow capacity.

5. The method according to claim 1 or 4, characterized in that, In step S2, the quantitative indicators characterize the rate of change of the cross-sectional area of ​​the river channel before and after treatment.

6. The method according to claim 1, characterized in that, In step S1, the GIS coordinates of all key nodes with deployed cameras are uploaded to construct a GIS topology network; In step S61, the first heavy rainfall process data of the key nodes associated with the abnormal river section are used as boundary conditions. Specifically, based on the GIS topology network, the two key nodes with cameras deployed upstream and downstream of the abnormal river section are selected; if multiple tributaries flow into the GIS topology network, the key nodes of the main tributary upstream of the confluence point are selected; the water level and surface velocity estimated by these key nodes in step S3 are used as boundary conditions.

Images