Panoramic detail two-channel video river channel monitoring method

Abstract

The invention relates to a panoramic detail two-channel video river channel monitoring method, which comprises the following steps of: 1, assembling a plurality of image sensors on a single camera, carrying out multi-sensor image splicing fusion, and outputting a spliced panoramic image; step 2, calibrating a panorama and a detail channel image based on the panorama image obtained by splicing and fusing the multiple sensors in the step 1; 3, based on a calibration result in the step 2, target behavior detection is carried out through the panoramic channel image; step 4, when there is an alarm, amplifying the detail channel image to the target position of the panoramic channel so as to continuously track the target; and 5, carrying out gauge-free water level monitoring based on the detail channel image. According to the invention, the panorama condition of the river channel can be monitored in time, and the arrangement and delivery cost is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of river channel monitoring, and relates to a river channel monitoring method, in particular to a panoramic-detail dual-channel video river channel monitoring method. Background Art

[0002] Traditional river monitoring cameras are mostly fixed-scene image monitoring; dome-type cameras need to rotate the pan / tilt to switch between multiple scene images to achieve multi-directional image monitoring, and cannot quickly observe the overall situation of the river at the same time; or multiple cameras need to be installed around the river to aggregate the monitoring images for overall river monitoring, which requires high deployment costs and a complex system.

[0003] Therefore, without solving the above technical problems, the present invention proposes a panoramic detail dual-channel video river monitoring method.

[0004] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a panoramic detail dual-channel video river monitoring method that can timely monitor the panoramic situation of the river and reduce the layout delivery cost.

[0006] The present invention solves the practical problem by adopting the following technical solutions:

[0007] A panoramic and detailed dual-channel video river monitoring method comprises the following steps:

[0008] Step 1: assemble multiple image sensors on a single camera, perform multi-sensor image stitching and fusion, and output a stitched panoramic image;

[0009] Step 2: Based on the panoramic image obtained by multi-sensor splicing and fusion in step 1, calibrate the panoramic and detail channel images;

[0010] Step 3: Based on the calibration result of step 2, target behavior detection is performed through the panoramic channel image;

[0011] Step 4: When there is an alarm, the detail channel image is enlarged to the target position of the panoramic channel, so as to continuously track the target;

[0012] Step 5: Perform water level monitoring without a water gauge based on the detail channel image.

[0013] Moreover, the specific steps of step 1 include:

[0014] (1) Assemble multiple image sensors on a single camera and create video stitching (AVS);

[0015] (2) Set the download splicing parameters;

[0016] (3) Determine the AVS offline input status

[0017] (4) If the answer in step (3) is no (offline), bind the splicing input relationship;

[0018] (5) If the answer in step (3) is yes (online), the original frame of the transmitted video is sent;

[0019] (6) Determine the AVS offline output status;

[0020] (7) If the answer of step (6) is no (offline), bind the splicing output relationship;

[0021] (8) If the answer in step (6) is yes (online), obtain the spliced ​​video frame;

[0022] (9) Perform image processing (cropping, distortion) according to parameters;

[0023] (10) Exit and release video splicing (AVS);

[0024] (11) Output the stitched panoramic image.

[0025] Moreover, the specific method of step 2 is:

[0026] Based on the panoramic image obtained by multi-sensor stitching and fusion in step 1, the panorama of the detail channel is controlled to rotate, and the cross position of the detail channel is marked on the panoramic channel image in a way of calibrating and marking the panoramic and detail feature points, thereby completing the calibration of the panoramic and detail channel images;

[0027] Moreover, the specific steps of step 3 include:

[0028] (1) Preprocessing the panoramic channel image;

[0029] (2) Build a deep learning detection model based on the RFCN target detection framework and perform training;

[0030] (3) Using the pre-trained detection model, the pre-processed panoramic channel image is detected based on the panoramic channel. The detection results output by the detection model include the coordinates of the circumscribed rectangular box of the human, vehicle, ship target and behavior, the category of the target and the corresponding confidence level, thereby completing the detection of the human, vehicle, ship target and behavior.

[0031] Moreover, the specific steps of performing water level monitoring without a water gauge based on the detail channel in step 5 include:

[0032] (1) Establishing a three-dimensional coordinate system: A three-dimensional coordinate system is established based on the river bank to describe the spatial position relationship between the camera and the test calibration object;

[0033] (2) Obtaining angle information: Calculate the angle information between the ball camera and the calibration point on the calibration point through the three-dimensional coordinate system;

[0034] The ordinate of the intersection is calculated as:

[0035] y=y1+(x-x1)*(y2-y1) / (x2-x1).

[0036] x1 is the x coordinate of calibration point 1;

[0037] y1 is the y coordinate of calibration point 1;

[0038] x2 is the x coordinate of calibration point 2;

[0039] y2 is the y coordinate of calibration point 2;

[0040] x is the x coordinate of the water level intersection point;

[0041] y is the y coordinate of the water level intersection point;

[0042] (3) Adjust the camera position: adjust the dome camera to the standby position and obtain the current angle information through the built-in tilt sensor;

[0043] (4) Calculating preset angle combinations: Calculating camera preset angle combinations that meet the preset water level area based on the acquired angle information and the positional relationship between the camera and the preset water level area;

[0044] Calculate the elevation of the water level intersection point as:

[0045] H=H1+(y–pixH1)*(H2–H1) / (pixH2–pixH1).

[0046] pixH1 is the pixel height of calibration point 1;

[0047] pixH2 is the pixel height of calibration point 2;

[0048] H1 is the elevation of calibration point 1;

[0049] H2 is the elevation of calibration point 2;

[0050] y is the y coordinate of the water level intersection point;

[0051] H is the elevation of the water level intersection point;

[0052] The calculated elevation of the water level intersection is the water level value, completing the water level monitoring.

[0053] (5) Storing mapping relationships: The mapping relationship between the camera preset angle combination and the preset water level area is pre-stored in the camera for subsequent water level conversion, thereby completing water level monitoring.

[0054] Advantages and beneficial effects of the present invention:

[0055] 1. The present invention proposes a panoramic-detail dual-channel video river monitoring method, which can improve the efficiency of real-time monitoring of rivers and reduce maintenance costs; through the linkage of panoramic and detail channel images, real-time and efficient monitoring of river water conditions can be carried out, thereby improving the efficiency of river warning and flood warning supervision; and the present invention can effectively monitor the behavior of people, vehicles, and ships around the river, identify the behavior and issue warnings, thereby reducing the maintenance cost of river monitoring; and the present invention supports all-weather monitoring of river water levels by establishing an elevation coordinate system without installing a water gauge, thereby improving the efficiency of real-time monitoring of rivers and the effect of intelligent identification and detection of water levels in rivers.

[0056] 2. The present invention arranges panoramic and detailed monitoring cameras, uses panoramic channel images to monitor the river and riverbank personnel, vehicles, ships and aircraft targets, and intelligently analyzes the target behavior; if there is an alarm or suspicious target, the detailed channel can be used to advance tracking; and it can identify whether there is sewage, floating objects, etc. on the river surface; the detailed channel is deployed to use a water gauge to detect the water level in real time and report to the platform, so as to realize the overall monitoring and supervision of the river. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the overlapping areas that need to be spliced ​​and cut according to the present invention;

[0058] Figure 2 It is a schematic diagram of pixel coordinates of a seam image of the present invention;

[0059] Figure 3 is a process flow chart of step 1 of the present invention;

[0060] Figure 4 The panoramic and detailed image calibration effect diagram of the present invention;

[0061] Figure 5 It is the panoramic and detailed image mapping effect diagram of the present invention;

[0062] Figure 6 is a process flow chart of step 3 of the present invention;

[0063] Figure 7 A schematic diagram of a coordinate system for calculating the angle between the dome camera and the calibration point of the present invention;

[0064] Figure 8 It is the effect diagram of the water-free ruler detection of the present invention. DETAILED DESCRIPTION

[0065] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0066] A panoramic and detailed dual-channel video river monitoring method comprises the following steps:

[0067] Step 1: assemble multiple image sensors on a single camera, perform multi-sensor image stitching and fusion, and output a stitched panoramic image;

[0068] In this embodiment, the image stitching and fusion in step 1 supports stitching together video contents of two or more image sensors, automatically weakening the seam difference and outputting a panoramic image; supports distortion removal and automatic cropping functions. Supports real-time fine-tuning parameter control. Ensures the panoramic video image effect of monitoring the river.

[0069] like Figure 1 As shown, it is a schematic diagram of the overlapping areas that need to be stitched and cropped. The multi-channel images are overlapped at the edges, and the fused bright bands are geometrically corrected and color processed through the fusion image technology, and finally a panoramic image without physical gaps is displayed.

[0070] like Figure 2 As shown, by judging the coordinate color of the image edge, the multi-channel image is overlapped at the edge. Figure 3 As shown, the specific steps of step 1 include:

[0071] (1) Assemble multiple image sensors on a single camera and create video stitching (AVS);

[0072] (2) Set the download splicing parameters;

[0073] (3) Determine the AVS offline input status

[0074] (4) If the answer in step (3) is no (offline), bind the splicing input relationship;

[0075] (5) If the answer in step (3) is yes (online), the original frame of the transmitted video is sent;

[0076] (6) Determine the AVS offline output status;

[0077] (7) If the answer of step (6) is no (offline), bind the splicing output relationship;

[0078] (8) If the answer in step (6) is yes (online), obtain the spliced ​​video frame;

[0079] (9) Perform image processing (cropping, distortion) according to parameters;

[0080] (10) Exit and release video splicing (AVS);

[0081] (11) outputting the stitched panoramic image;

[0082] Step 2: Based on the panoramic image obtained by multi-sensor splicing and fusion in step 1, calibrate the panoramic and detail channel images;

[0083] The specific method of step 2 is:

[0084] Based on the panoramic image obtained by multi-sensor stitching and fusion in step 1, the panorama of the detail channel is controlled to rotate, and the cross position of the detail channel is marked on the panoramic channel image in a way of calibrating and marking the panoramic and detail feature points, thereby completing the calibration of the panoramic and detail channel images;

[0085] This is for subsequent panoramic recognition and detection of targets or for tracking of detailed channel images after an emergency situation.

[0086] like Figure 4 and Figure 5 As shown, the 2 specific steps include:

[0087] (1) Control the detail channel gimbal and fix the image to a random position;

[0088] (2) Mark the cross position of the detail channel on the panoramic channel image;

[0089] (3) Repeat steps (1) and (2) to complete the calibration operation;

[0090] In this embodiment, after controlling the pan / tilt of the detail channel, the water level / vertical / zoom (PTZ) and the width and height of the image are mapped to the x, y resolution coordinates of the panoramic channel image; when the panoramic channel detects and identifies the target or there is an alarm, the pan / tilt (PTZ) of the detail channel can be rotated to achieve target tracking and clearer monitoring of the target situation.

[0091] Step 3: Based on the calibration result of step 2, target behavior detection is performed through the panoramic channel image;

[0092] The specific steps of step 3 include:

[0093] (1) Preprocessing the panoramic channel image;

[0094] (2) Build a deep learning detection model based on the RFCN target detection framework and perform training;

[0095] (3) Using the pre-trained detection model, the pre-processed panoramic channel image is detected based on the panoramic channel. The detection results output by the detection model include the coordinates of the circumscribed rectangular box of the human, vehicle, ship target and behavior, the category of the target and the corresponding confidence level, thereby completing the detection of the human, vehicle, ship target and behavior.

[0096] Step 4: When there is an alarm, the detail channel image is enlarged to the target position of the panoramic channel, so that the target can be tracked continuously.

[0097] In this embodiment, target behavior detection of panoramic or detail channel images is implemented based on the deep learning network architecture constructed based on RFCN, which supports detection and recognition of human, vehicle, and ship targets and behaviors;

[0098] Determine whether there is an alarm based on the movement trajectory of people, vehicles and ships. For example, an alarm will be triggered after entering a river area or hitting a line, and the detail channel can be linked to zoom in and advance the tracking target; the alarm can be reported to the platform for processing.

[0099] Among them, the specific steps of the method for building a deep learning network architecture based on RFCN include:

[0100] The ResNet-18 network is used as the basic network for image feature extraction. The feature map output by the basic network is input into the RFCN target detection framework to finally realize the detection and recognition of target behavior.

[0101] Among them, the size and proportion of the anchors should try to meet the scale and proportion range of the target in the sample set, so that the network can use the position (x, y, w, h) of the prediction box, the confidence, the position and category of the true value to calculate the loss, and minimize the final loss through repeated iterative calculations.

[0102] Training the model to detect people, cars, and ships and output the detection results mainly includes using the pre-trained model to detect the image to be detected. The detection results output by the model include the coordinates of the target's circumscribed rectangular box, the category to which the target belongs, and the corresponding confidence level. The specific process is as follows: Figure 6 shown.

[0103] Step 5: Perform water level monitoring without a water gauge based on the detail channel image.

[0104] In this embodiment, the detail channel can read the readings of the water gauge installed in the river, or monitor the water level through the water gauge-free measurement technology based on visual intelligent analysis, so as to achieve real-time and all-weather effective monitoring of the water level in the river.

[0105] The specific steps of performing water level monitoring without a water gauge based on the detail channel in step 5 include:

[0106] (1) Establishing a three-dimensional coordinate system: A three-dimensional coordinate system is established based on the river bank to describe the spatial position relationship between the camera and the test calibration object;

[0107] (2) Obtaining angle information: Calculate the angle information between the ball camera and the calibration point on the calibration point through the three-dimensional coordinate system;

[0108] like Figure 7 As shown, the ordinate of the intersection is calculated as:

[0109] y=y1+(x-x1)*(y2-y1) / (x2-x1).

[0110] x1 is the x coordinate of calibration point 1;

[0111] y1 is the y coordinate of calibration point 1;

[0112] x2 is the x coordinate of calibration point 2;

[0113] y2 is the y coordinate of calibration point 2;

[0114] x is the x coordinate of the water level intersection point;

[0115] y is the y coordinate of the water level intersection point;

[0116] (3) Adjust the camera position: adjust the dome camera to the standby position and obtain the current angle information through the built-in tilt sensor;

[0117] (4) Calculating preset angle combinations: Calculating camera preset angle combinations that meet the preset water level area based on the acquired angle information and the positional relationship between the camera and the preset water level area;

[0118] Calculate the elevation of the water level intersection point as:

[0119] H=H1+(y–pixH1)*(H2–H1) / (pixH2–pixH1).

[0120] pixH1 is the pixel height of calibration point 1;

[0121] pixH2 is the pixel height of calibration point 2;

[0122] H1 is the elevation of calibration point 1;

[0123] H2 is the elevation of calibration point 2;

[0124] y is the y coordinate of the water level intersection point;

[0125] H is the elevation of the water level intersection point;

[0126] The calculated elevation of the water level intersection is the water level value, completing the water level monitoring.

[0127] (5) Storing mapping relationships: The mapping relationship between the camera preset angle combination and the preset water level area is pre-stored in the camera for subsequent water level conversion, thereby completing water level monitoring.

[0128] like Figure 8 The figure shown is the effect diagram when there is no water gauge detection.

[0129] The working principle of the present invention is:

[0130] With the development and maturity of AI intelligent recognition, AI big models, AI images and other technologies and smart chips in recent years, multiple image sensors can be installed on a single camera for image fusion, so as to achieve panoramic image monitoring of the river at the same time; cameras with high-performance hardware can add detailed video channels that support pan-tilt control, and can perform image magnification and tracking monitoring of alarm situations and target locations monitored by panoramic images.

[0131] The panoramic and detail channels can be set up with multiple monitoring tasks at the same time, such as the detail channel identifying the water level and tracking the target, and the panoramic channel monitoring floating objects, illegal mining and unloading, illegal fishing, illegal sewage discharge, water color, etc. It solves the problems of timely monitoring the panoramic situation of the river and reducing the layout and delivery costs.

[0132] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A panoramic and detailed dual-channel video river monitoring method, characterized by: The following steps are involved: Step 1: assemble multiple image sensors on a single camera, perform multi-sensor image stitching and fusion, and output a stitched panoramic image; Step 2: Based on the panoramic image obtained by multi-sensor splicing and fusion in step 1, calibrate the panoramic and detail channel images; Step 3: Based on the calibration result of step 2, target behavior detection is performed through the panoramic channel image; Step 4: When there is an alarm, the detail channel image is enlarged to the target position of the panoramic channel, so as to continuously track the target; Step 5: Perform water level monitoring without a water gauge based on the detail channel image.

2. A panoramic and detailed dual-channel video river monitoring method according to claim 1, characterized in that: The specific steps of step 1 include: (1) Assemble multiple image sensors on a single camera and create video stitching (AVS); (2) Set the download splicing parameters; (3) Determine the AVS offline input status (4) If the answer in step (3) is no (offline), bind the splicing input relationship; (5) If the answer in step (3) is yes (online), the original frame of the transmitted video is sent; (6) Determine the AVS offline output status; (7) If the answer of step (6) is no (offline), bind the splicing output relationship; (8) If the answer in step (6) is yes (online), obtain the spliced ​​video frame; (9) Perform image processing (cropping, distortion) according to parameters; (10) Exit and release video splicing (AVS); (11) Output the stitched panoramic image.

3. The method for monitoring a river with a panoramic and detailed dual-channel video according to claim 1, characterized in that: The specific method of step 2 is: Based on the panoramic image obtained by multi-sensor stitching and fusion in step 1, the pan / tilt of the detail channel is controlled to mark the cross position of the detail channel on the panoramic channel image in a way of calibrating and marking the panoramic and detail feature points, thereby completing the calibration of the panoramic and detail channel images.

4. The method for monitoring a river with a panoramic and detailed dual-channel video according to claim 1, characterized in that: The specific steps of step 3 include: (1) Preprocessing the panoramic channel image; (2) Build a deep learning detection model based on the RFCN target detection framework and perform training; (3) Using the pre-trained detection model, the pre-processed panoramic channel image is detected based on the panoramic channel. The detection results output by the detection model include the coordinates of the circumscribed rectangular box of the human, vehicle, ship target and behavior, the category of the target and the corresponding confidence level, thereby completing the detection of the human, vehicle, ship target and behavior.

5. The method for river monitoring with panoramic and detailed dual-channel video according to claim 1, characterized in that: The specific steps of performing water level monitoring without a water gauge based on the detail channel in step 5 include: (1) Establishing a three-dimensional coordinate system: A three-dimensional coordinate system is established based on the river bank to describe the spatial position relationship between the camera and the test calibration object; (2) Obtaining angle information: Calculate the angle information between the ball camera and the calibration point on the calibration point through the three-dimensional coordinate system; The ordinate of the intersection is calculated as: y=y1+(x-x1)*(y2-y1) / (x2-x1); x1 is the x coordinate of calibration point 1; y1 is the y coordinate of calibration point 1; x2 is the x coordinate of calibration point 2; y2 is the y coordinate of calibration point 2; x is the x coordinate of the water level intersection point; y is the y coordinate of the water level intersection point; (3) Adjust the camera position: adjust the dome camera to the standby position and obtain the current angle information through the built-in tilt sensor; (4) Calculating preset angle combinations: Calculating camera preset angle combinations that meet the preset water level area based on the acquired angle information and the positional relationship between the camera and the preset water level area; Calculate the elevation of the water level intersection point as: H=H1+(y–pixH1)*(H2–H1) / (pixH2–pixH1); pixH1 is the pixel height of calibration point 1; pixH2 is the pixel height of calibration point 2; H1 is the elevation of calibration point 1; H2 is the elevation of calibration point 2; y is the y coordinate of the water level intersection point; H is the elevation of the water level intersection point; The calculated elevation of the water level intersection is the water level value, completing the water level monitoring; (5) Storing mapping relationships: The mapping relationship between the camera preset angle combination and the preset water level area is pre-stored in the camera for subsequent water level conversion, thereby completing water level monitoring.

Images