Video monitoring and dam safety monitoring system for reservoirs

Abstract

The embodiment of the application discloses a reservoir-oriented video monitoring and dam safety monitoring system, and is applied to the water conservancy field. The system comprises a camera cluster, a monitoring module, an alarm module and a record management module. The monitoring module is used for acquiring at least one video data. For any one video data, in response to the case that a target frame in which a target object is detected exists in the video data, the target object is subjected to gradient-based optical flow fusion image feature continuity object detection, and the target frame and the monitoring result are fed back to the alarm module. The alarm module is used for performing ship feature extraction on the target frame and the monitoring result, performing ship identification based on the ship feature extraction result, and performing alarm according to the ship identification result. The record management module is used for generating a monitoring record based on the ship identification result and the ship feature extraction result. The application can be implemented in real time.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of water conservancy, and in particular to a video surveillance and dam safety monitoring system for reservoirs. Background Art

[0002] Reservoirs are frontiers for flood control and drought relief, addressing the uneven spatial and temporal distribution of water resources, and fully developing and utilizing them. Digital reservoirs, virtual counterparts of prototype reservoirs, build on traditional technologies by leveraging modern technologies like the internet, big data, and artificial intelligence. Focusing on core tasks such as project safety, operational management, and forecasting and scheduling, they integrate digital management systems with features like automatic perception, scene visualization, intelligent analysis, unified standards, and efficient collaboration. Achieving unmanned, real-time, and intelligent reservoir management is a key component of digital reservoir development. Summary of the Invention

[0003] In order to solve at least one of the above technical problems, an embodiment of the present application provides a video surveillance and dam safety monitoring system for a reservoir.

[0004] In one aspect, an embodiment of the present application provides a video surveillance and dam safety monitoring system for a reservoir, the system comprising a camera cluster, a monitoring module, an alarm module, and a record management module. The camera cluster comprises multiple cameras, and the video data captured by each camera forms a full field of view covering the target area where the reservoir and the dam are located.

[0005] The monitoring module is configured to acquire at least one channel of video data, any of which is captured by a corresponding camera in the camera cluster, the corresponding camera being configured to continuously capture the reservoir or dam; and, for any of the video data, in response to the presence of a target frame in which a target object is detected in the video data, perform persistent object detection on the target object based on the video data by fusing gradient-based optical flow and image features, and feed back the target frame and monitoring results to the alarm module;

[0006] The alarm module is configured to extract vessel features from both the target frame and the monitoring result, identify the vessel based on the vessel feature extraction result, and issue an alarm based on the vessel identification result; and transmit the vessel identification result and the vessel feature extraction result to the record management module;

[0007] The record management module is used to generate a monitoring record based on the ship identification result and the ship feature extraction result.

[0008] In one embodiment, the target object is an object identified as a vessel, the target frame is a frame in the video data in which the target object is first detected within a first time interval, and the monitoring module is configured to perform the following operations:

[0009] Sequentially extracting each initial image frame from the video data within the first time interval;

[0010] For each initial image frame, the following first operation is performed: performing channel longitudinal connection processing on the initial image frame and a preset reference frame to obtain a target image frame, where the preset reference frame is an image frame or a fusion result thereof captured by the camera corresponding to the video data without capturing any objects that do not belong to the target area, and the preset reference frame has the same clarity, field of view, and number of channels as the initial image frame; and detecting the target image frame based on a first object detection model; wherein the first object detection model is a lightweight object detection model for detecting illegally entered objects, the first object detection model is trained to classify object types, and the classification accuracy of the first object detection model is lower than a first threshold and the running speed is higher than a second threshold;

[0011] In response to the first detection of a target object, the initial image frame where the target object is located is determined as the target frame, and the first operation is stopped.

[0012] In one embodiment, the monitoring module is further configured to perform the following operations:

[0013] During the execution of the first operation, in response to detecting an illegal entry of an object other than the target object, transmitting an initial image frame containing the object to the alarm module, triggering an alarm of a prompt level of the alarm module;

[0014] In response to the first detection of the target object, the target frame is fed back to the alarm module to trigger an alarm of an alarm level of the alarm module.

[0015] In one embodiment, the performing persistent object detection on the target object based on the video data by fusing gradient-based optical flow and image features includes:

[0016] For other image frames in the video data within the first time interval that are located after the initial image frame where the target object is located, the second operation is performed in sequence:

[0017] The second operation is specifically:

[0018] Performing gradient-based optical flow object mapping on all target objects in a frame previous to the current frame, and obtaining an object detection result of the current frame based on the mapping result, wherein the object detection result includes all target objects in the current frame;

[0019] Obtaining a frame identifier of the current frame and a frame identifier of the target frame;

[0020] In response to a situation where a remainder of the frame identifier of the target frame relative to the frame identifier of the current frame is zero, performing object detection based on image features on the current frame, and updating an object detection result of the current frame according to the detection result;

[0021] In response to the situation that all target objects in the object detection result of the current frame are not completely the same as all target objects in the previous frame, the current frame is determined as the monitoring result.

[0022] In one embodiment, extracting vessel features from both the target frame and the monitoring result includes:

[0023] The target frame and each image in the monitoring result are respectively input into a second object detection model for vessel feature extraction; wherein the second object detection model is a heavyweight information extraction model, the second object detection model is trained as a model for vessel feature extraction, and the extraction accuracy of the second object detection model is higher than a third threshold and the operating speed is lower than a fourth threshold.

[0024] In one embodiment, the second object detection model is trained by the following method:

[0025] Acquire a vessel sample image set, wherein each sample sequence in the vessel sample image set includes a plurality of vessel sample images and corresponding annotation information, and each of the plurality of vessel sample images includes the same vessel;

[0026] For any sample sequence, each sample image in the sample sequence is independently input into a first feature extraction model and a second feature extraction model, thereby obtaining a first sample information matrix sequence and a second sample information matrix sequence, respectively. The first feature extraction model is a pre-trained model for extracting overall features of a ship, and the second feature extraction model is a pre-trained model for extracting features of key areas of a ship.

[0027] performing downsampling processing on each first sample information matrix in the first sample information matrix sequence to obtain a first sample information distribution;

[0028] performing downsampling processing on each second sample information matrix in the second sample information matrix sequence to obtain a second sample information distribution;

[0029] Inputting the first sample information matrix sequence into a pre-trained ship detection model to obtain a first detection result distribution;

[0030] Inputting the second sample information matrix sequence into a pre-trained ship detection model to obtain a second detection result distribution;

[0031] Obtaining a heterogeneous model ontology loss based on a difference between the first sample information distribution and the second sample information distribution;

[0032] Obtaining a different model performance loss based on a difference between the first detection result distribution and the second detection result distribution;

[0033] Obtaining a first ship detection loss based on a difference between the first detection result distribution and the annotation information corresponding to the sample sequence;

[0034] obtaining a second ship detection loss based on a difference between the second detection result distribution and the annotation information corresponding to the sample sequence;

[0035] determining a model training loss according to the heterogeneous model ontology loss, the heterogeneous model performance loss, the first ship detection loss, and the second ship detection loss;

[0036] Parameters of the first feature extraction model and parameters of the second extraction model are adjusted based on the model training loss.

[0037] In one embodiment, after training, a joint model is obtained based on the first feature extraction model obtained after adjusting parameters and the ship detection model, and the joint model is used as the second object detection model.

[0038] In one embodiment, extracting ship features from both the target frame and the monitoring result includes: extracting ship features from both the target frame and the monitoring result using a first feature extraction model in the second object detection model;

[0039] The performing ship identification based on the ship feature extraction result includes: inputting the ship feature extraction result into a ship detection model in the second object detection model to perform ship identification.

[0040] The embodiment of the present application provides a video surveillance and dam safety monitoring system for reservoirs. By deploying cameras in the target area where the reservoir and dam are located, each location in the target area that needs to be automatically monitored can be automatically photographed. By analyzing the video data obtained from the photography, any object that does not belong to the target area and illegally enters the target area can be automatically discovered, and different alarm strategies can be automatically triggered according to the type of object. The embodiment of the present application effectively decomposes the supervision task for the target area by designing multiple models with different focuses, thereby solving the real-time problem. In other words, the embodiment of the present application can achieve real-time and high-intensity supervision. Specifically, for objects that are not very important and illegally enter the target area, real-time discovery and real-time alarm can be achieved. For objects that are very important and illegally enter the target area, discovery, real-time alarm, real-time tracking, object detailed information mining and recording can be achieved. Thus, scientific supervision is completed in a hierarchical and situation-specific manner. However, it should be noted that no matter what the situation, the present application can achieve real-time control. This is due to the advantages of the design of the model and the system architecture design in the embodiment of the present application. By achieving functional decoupling and uniform distribution of load, the various components of the system perform their respective functions, and the load is balanced, and the operating performance of the components is balanced. There is no situation where the overall system response speed is slowed down. This is the key to achieving real-time control in the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 This is a logical block diagram of a reservoir-oriented video surveillance and dam safety monitoring system provided in an embodiment of this specification;

[0043] Figure 2 This is a schematic diagram of the electrical component load provided in an embodiment of the present application;

[0044] Figure 3 This is a second operation flow chart provided in an embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of gradient-based optical flow object mapping provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0047] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not necessarily have to include all the steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0048] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0049] Hereinafter, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more features. In the description of the present embodiment, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0050] Before the detailed description of the present application, it must be emphasized that the contents in the present application are implemented with the authorization of the relevant departments or users, and must be applied after sufficient authorization in the future.

[0051] The reservoir-oriented video monitoring and dam safety monitoring system of the present application is introduced as follows, Figure 1 A schematic diagram of the reservoir-oriented video monitoring and dam safety monitoring system provided by the present application is shown.

[0052] The system includes a camera cluster, a monitoring module, an alarm module and a record management module. The camera cluster includes multiple cameras, and the full field of view formed by the video data captured by each camera covers the target area where the reservoir and the dam are located.

[0053] The target area can be understood as an area requiring supervision, including reservoirs and dams. Specifically, cameras can be placed at relevant locations within the target area based on actual monitoring needs. Each camera can continuously capture and output video data. Of course, the location of the target area and the placement of each camera are determined based on actual reservoir and dam supervision requirements and are not the focus of the embodiments of this application.

[0054] The monitoring module is used to obtain at least one channel of video data, any of which is captured by a corresponding camera in the camera cluster, and the corresponding camera is configured to continuously capture the reservoir or dam; and, for any channel of video data, in response to the presence of a target frame in which a target object is detected in the video data, a continuous object detection based on a fusion of gradient-based optical flow and image features is performed on the target object based on the video data, and the target frame and the monitoring result are fed back to the alarm module. Taking into account the actual regulatory needs of reservoirs and dams, in the embodiment of the present application, objects that illegally enter the target area are divided into two categories, ships and other objects. The illegal entry of ships may involve safety issues of reservoirs and dams. Therefore, stricter regulatory measures are adopted, while for other objects, such as drifting objects, relatively loose regulatory measures can be adopted. Strict regulatory measures and loose regulatory measures are detailed below.

[0055] Similarly, in the embodiments of this application, the target object refers to an object classified as a vessel. Continuous object detection indicates that the embodiments of this application continuously detect illegally entering vessels. This allows for real-time tracking of the movements of these vessels, allowing for close monitoring and robust oversight of these vessels. The target frame and monitoring results can be considered captured images of the illegally entering vessel. Feeding these images to the alarm module facilitates the module's ability to generate alerts, extract highly detailed information, perform object detection, and further generate relevant records, thereby enabling robust oversight.

[0056] The alarm module is configured to extract vessel features from the target frame and the monitoring result, identify the vessel based on the vessel feature extraction result, and generate an alarm based on the vessel identification result; and transmit the vessel identification result and the vessel feature extraction result to the record management module. The record management module is configured to generate a monitoring record based on the vessel identification result and the vessel feature extraction result.

[0057] As can be seen from the foregoing, the target object is an object identified as a ship, the target frame is a frame in the video data in which the target object is first detected within the first time interval, and the monitoring module is configured to perform the following operations:

[0058] (1) Sequentially extracting each initial image frame from the video data within the first time interval.

[0059] It should be noted that the embodiment of the present application does not limit the first time interval. For example, every 10 minutes can be determined as a first time interval. The embodiment of the present application describes one of the first time intervals, and the same applies to the other first time intervals. Of course, the operations of steps (1) to (2) are performed for each channel of video data. The embodiment of the present application describes the operation for one channel of video data.

[0060] (2) performing the following first operation for each initial image frame: performing channel longitudinal connection processing on the initial image frame and a preset reference frame to obtain a target image frame, wherein the preset reference frame is an image frame or a fusion result thereof captured by the camera corresponding to the video data without capturing any object that does not belong to the target area, and the preset reference frame has the same clarity, field of view and number of channels as the initial image frame; and detecting the target image frame based on a first object detection model; wherein the first object detection model is a lightweight object detection model for detecting illegally entered objects, the first object detection model is trained as a model for object type classification, and the classification accuracy of the first object detection model is lower than a first threshold and the running speed is higher than a second threshold.

[0061] The present embodiment emphasizes that the preset reference frame is an image frame or a fusion result of the camera corresponding to the video data when no object outside the target area is captured, and the preset reference frame has the same clarity, field of view, and number of channels as the initial image frame. Since the present embodiment needs to detect any object that illegally enters the target area, theoretically, illegal entry is a type of abnormal loading that cannot be controlled by humans. No one can know in advance what type of object has illegally entered the target area, and no prior information can be obtained. This makes it difficult to determine which objects in the initial image frame should be present and which objects have entered illegally. To solve this technical problem, the present embodiment uses a preset reference frame. The preset reference frame is a picture or a fusion result of the picture captured when it is determined that no foreign object has illegally entered. The picture contains sufficient background information. Specifically, multiple such pictures can be fused to obtain the preset reference frame. This allows the preset reference frame to fully include information about objects that should be present. Of course, objects that are not in the preset reference frame but in the initial image frame are likely to be objects that have entered illegally. The samples used in the training process of the first object detection model are based on the same concept. The samples are also obtained by longitudinally connecting the channels of the image that needs to be detected for illegal entry and the reference image containing information about the object that should have existed. Therefore, the first object detection model has the ability to detect illegally entered objects. Of course, the training process is not limited. The particularity of the sample is the key that needs to be emphasized in the embodiments of this application. For the image that needs to be detected for illegal entry (image 1), it has three channels {A1, B1, C1}. For the reference image (image 2) containing information about the object that should have existed, it has three channels {A2, B2, C2}. Then the longitudinal connection of the channels means that the sample has three channels {A1+A2, B1+B2, C1+C2}.

[0062] In order to make the preset reference frame include as much information as possible about the objects that should have existed, the images taken under various circumstances without illegal entry of foreign objects can be fused, and the fusion results can be adjusted for clarity, field of view, and number of channels to obtain the preset reference frame of step (2). For example, image 1, image 2, and image 3 are all images without illegal entry of foreign objects. Image 1 has a bird, image 2 has a water level detection pile protruding from the reservoir, and image 3 has a newly built monitoring hut. By fusing image 1, image 2, and image 3, a preset reference frame including the bird, water level detection pile, and monitoring hut can be obtained. This ensures that in the subsequent detection process, the bird, water level detection pile, and monitoring hut will not be considered as objects that have illegally entered the target area and will not trigger an alarm.

[0063] Please refer to Figure 2 , which shows a schematic diagram of the load of the electrical components in the present application. For each initial image frame on which the first operation is performed, detection needs to be implemented based on the first object detection model. Obviously, the number of image frames that need to be detected is likely to be large, and it is also likely to be continuous. This requires that the first object detection model for implementing the detection must have a good detection speed. In order to meet the detection speed requirements, the embodiment of the present application adjusts the scale of the first object detection model and ensures the detection speed by reducing the detection accuracy. In other words, the first object detection model only needs to be able to identify objects that illegally enter the target area and roughly divide them into ships and other objects to meet the requirements of the embodiment of the present application. Obviously, the embodiment of the present application only requires the first object detection model to implement a basic classification task. The design of the first object detection model focuses on speed requirements and appropriately sacrifices accuracy requirements.

[0064] Specifically, the first object detection model is a lightweight object detection model used to detect illegally entered objects. The first object detection model is trained to classify object types. The classification accuracy of the first object detection model is lower than a first threshold and the operating speed is higher than a second threshold. This embodiment of the application does not limit the first and second thresholds. However, it is certain that the first threshold is necessarily lower than the third threshold used to limit the ship feature extraction accuracy of the heavyweight information extraction model described below, and the second threshold is higher than the fourth threshold used to limit the operating speed of the heavyweight information extraction model described below. Of course, the settings of the third and fourth thresholds are also determined based on actual conditions.

[0065] As mentioned above, the first object detection model only implements a basic classification task. The first object detection model can be obtained by training a neural network. This training method is also very simple and can refer to other existing technologies. Of course, in order to meet the operating requirements of the embodiment of the present application, relevant personnel can also perform operations such as model pruning to further reduce the scale of the first object detection model. Of course, model pruning will also sacrifice accuracy, but the embodiment of the present application has low accuracy requirements for the first object detection model. This is due to the design framework of the electrical component load of the present application. Because the task of fully automatic reservoir dam supervision is large and heavy, and the real-time requirements are very high, the use of a single model will inevitably not be able to meet the task execution requirements and real-time requirements at the same time. Therefore, the embodiment of the present application decomposes the task. The first object detection model only needs to perform basic classification tasks. It will be frequently started and run for a long time. The number of detected images is also large. Therefore, it is configured to perform basic classification tasks in order to meet the real-time requirements as much as possible.

[0066] (3) In response to the first detection of the target object, the initial image frame where the target object is located is determined as the target frame, and the first operation is stopped.

[0067] If a ship is detected, the target frame is determined. At this point, the first operation can be stopped and the second operation can be triggered. The second operation will be described in detail below and is not detailed here. The frame ID of the target frame can be determined to be a preset non-zero value, such as 5, which is the frame ID sequence number of the frames following the target frame.

[0068] Furthermore, during the execution of the first operation, in response to detecting an object other than the target object that has illegally entered, the initial image frame containing the object is transmitted to the alarm module, triggering an alert-level alarm in the alarm module. Of course, in response to the initial detection of the target object, the target frame is fed back to the alarm module, triggering an alert-level alarm in the alarm module. The former is an indication level, while the latter is an alarm level. The alarm method, content, and frequency differ, but the specific differences can be set based on actual circumstances and are not detailed in this embodiment. This also reflects the different regulatory approaches for illegally entering vessels and non-vessels: strict regulation of the former and relatively relaxed regulation of the latter.

[0069] For other image frames located after the initial image frame where the target object is located in the video data within the first time interval, the second operation is performed in sequence. In the embodiment of the present application, the second operation is as follows: Figure 3 As shown, the second operation is specifically:

[0070] Step 1: Perform gradient-based optical flow object mapping on all target objects in the previous frame of the current frame, and obtain an object detection result of the current frame based on the mapping result, wherein the object detection result includes all target objects in the current frame.

[0071] Gradient-based optical flow object mapping refers to mapping all target objects in the object detection results corresponding to the previous frame of the current frame to the current frame, which essentially realizes the traceability of all target objects. Figure 4 As shown, it shows a schematic diagram of gradient-based optical flow object mapping. The left picture is the previous frame of the current frame, which includes two target objects, namely object 1 and object 2 located at position A and position B respectively. Gradient-based optical flow object mapping is performed on these two target objects. It can be found that object 1 and object 2 are located at position C and position D of the current frame respectively. This not only obtains the trajectory but also tries not to lose the target object. Gradient-based optical flow object mapping is a prior art, and the embodiments of this application do not limit this. This technology is also currently fast and will not affect real-time supervision due to the operation of this technology.

[0072] Step 2: Obtain the frame identifier of the current frame and the frame identifier of the target frame.

[0073] Step 3: In response to the situation that the remainder of the frame identifier of the target frame relative to the frame identifier of the current frame is zero, perform object detection based on image features on the current frame, and update the object detection result of the current frame according to the detection result.

[0074] However, the embodiment of the present application also believes that the reliability of the gradient-based optical flow object mapping is also limited. The target objects that have been detected may still be lost, and the newly appeared target objects may not be found. Therefore, the embodiment of the present application sets the case where the frame identifier of the target frame has a remainder of zero relative to the frame identifier of the current frame, and performs object detection based on image features on the current frame, and updates the object detection result of the current frame according to the detection result. In this way, the lost target objects and the newly appeared target objects can be found in time, thereby ensuring timely and effective supervision. It should be noted that object detection based on image features is only required when the frame identifier of the target frame has a remainder of zero relative to the frame identifier of the current frame. Of course, the present application does not limit the model of object detection, which can be obtained using existing technology. Image feature-based object detection is only required when the remainder of the frame identifier of the target frame relative to the frame identifier of the current frame is zero, which can reduce the startup frequency of image feature-based object detection. That is, the embodiment of the present application reduces the startup frequency of image feature-based object detection by introducing gradient-based optical flow object mapping. Image feature-based object detection is slightly slower than gradient-based optical flow object mapping. Therefore, this design can not only meet real-time detection requirements, but also promptly detect lost target objects and newly appeared target objects, thereby ensuring timely and effective supervision. It is another unique design of the embodiment of the present application to meet real-time requirements.

[0075] Step 4: In response to the situation that all target objects in the object detection result of the current frame are not completely the same as all target objects in the previous frame, the current frame is determined as the monitoring result.

[0076] That is to say, if all target objects do not change, only the target frame needs to be transmitted to the alarm module for processing. If there is a change, more or less, only the target frame and the changed image need to be transmitted. It is not necessary to transmit all images with target objects to the alarm module, thereby reducing the amount of data processed by the alarm module and further speeding up the system to meet the real-time requirements.

[0077] Furthermore, in the embodiment of the present application, the target frame and each image in the monitoring result are respectively input into a second object detection model for vessel feature extraction; wherein the second object detection model is a heavyweight information extraction model, the second object detection model is trained to perform vessel feature extraction, and the extraction accuracy of the second object detection model is higher than a third threshold and the operating speed is lower than a fourth threshold. Please refer to Figure 2 The heavyweight information extraction model is a refined model that can achieve very fine information extraction and accurate classification. Therefore, its speed is relatively slow. It is used in strict supervision to obtain richer information for supervision. In the embodiment of the present application, only the target frames and monitoring results need to be processed by the second object detection model. In other words, the startup frequency of the second object detection model is also very low, which can meet the real-time requirements of the system.

[0078] In order to ensure that the accuracy of ship feature extraction of the second object model fully meets the needs of reservoir supervision, the embodiment of the present application details the training method of the model. Of course, the embodiment of the present application does not limit the specific structure of the second object model, because in theory any neural network that can adapt to the training method can be used. Therefore, the embodiment of the present application does not limit the structure.

[0079] In a specific embodiment, the second object detection model is trained by the following method:

[0080] Step 10: Acquire a set of sample ship images, wherein each sample sequence in the set of sample ship images includes a plurality of sample ship images and corresponding annotation information, and each of the plurality of sample ship images includes the same ship.

[0081] The marking information may include the location of the same vessel and the identification of the same vessel.

[0082] Step 11: For any sample sequence, each sample image in the sample sequence is independently input into the first feature extraction model and the second feature extraction model, and the first sample information matrix sequence and the second sample information matrix sequence are obtained respectively, wherein the first feature extraction model is a pre-trained model for extracting the overall features of the ship, and the second feature extraction model is a pre-trained model for extracting the features of the key areas of the ship.

[0083] The embodiments of the present application do not limit the structure of the first feature extraction model and the second feature extraction model, as long as they can perform information extraction, they can be, for example, pyramid networks, deep convolutional networks, etc. Each sample image can be output with corresponding first sample information matrices and second sample information matrices after passing through the first feature extraction model and the second feature extraction model. Of course, in order to improve the training speed, the first feature extraction model is a pre-trained model for extracting the overall features of the ship, and the second feature extraction model is a pre-trained model for extracting the features of the key areas of the ship. There is no limitation on the pre-training method and key positions. Pre-training can be carried out under the guidance of the parameters of the open source network. The key positions are related to regulatory requirements and can be set according to actual conditions. The pre-training process adopts existing technology, which is not limited by the embodiments of the present application.

[0084] Step 12: Perform downsampling processing on each first sample information matrix in the first sample information matrix sequence to obtain a first sample information distribution.

[0085] The embodiments of the present application do not limit the downsampling method. Reference can be made to the prior art. Downsampling can be used to reduce the dimension of the first sample information matrix sequence. For example, each element of the first sample information matrix sequence is mapped to a multidimensional vector, thereby obtaining a multidimensional vector sequence. The multidimensional vector sequence can represent the first sample information distribution.

[0086] Step 13: Perform downsampling processing on each second sample information matrix in the second sample information matrix sequence to obtain a second sample information distribution.

[0087] Step 14: Input the first sample information matrix sequence into the pre-trained ship detection model to obtain a first detection result distribution.

[0088] Step 15: Input the second sample information matrix sequence into the pre-trained ship detection model to obtain a second detection result distribution.

[0089] The purpose of feature extraction is to extract sufficiently accurate information to achieve precise ship detection. In other words, the accuracy of the ship detection results directly measures the accuracy of the extracted information. Driven by the idea of ​​results, the present embodiment introduces a pre-trained ship detection model. This model processes the first sample information matrix sequence and the second sample information matrix sequence separately to obtain a first detection result distribution and a second detection result distribution. Taking the first detection result distribution as an example, inputting the ship detection model into each matrix in the first sample information matrix sequence yields the corresponding detection result. The distribution formed by each detection result is the first detection result distribution. The ship detection model of the present application is only used to measure the quality of sample information and thus guide model training. Therefore, there are no requirements for the ship detection model itself; it can be used for ship detection and can be trained by itself or obtained from an open source network.

[0090] Step 16: Obtain a heterogeneous model ontology loss based on the difference between the first sample information distribution and the second sample information distribution.

[0091] The embodiment of the present application uses hetero-model ontology loss to describe the distribution difference of information extracted by different models (first feature extraction model and second feature extraction model) when designing loss. The embodiment of the present application believes that for the same sample sequence, it is normal for the first feature extraction model and the second feature extraction model to extract different information due to different focus positions, but the vertical distribution of information in the sample sequence should be consistent. If it is inconsistent, hetero-model ontology loss will occur. Of course, the present application does not limit the specific method of characterizing various "differences", because there are a large number of methods for characterizing various "differences" in the prior art. The focus of this application and one of the invention points is to disclose "what kind of differences" and which differences must be characterized to achieve the training purpose, rather than how the specific differences are characterized.

[0092] Step 17: Obtain a different model performance loss based on the difference between the first detection result distribution and the second detection result distribution.

[0093] The embodiment of the present application uses heterogeneous model performance loss when performing loss design to describe the difference in ship detection results caused by the difference in the distribution of information extracted by different models (the first feature extraction model and the second feature extraction model). Because the same ship detection model is used, the difference in the results indicates a further description of the heterogeneous model difference between the first feature extraction model and the second feature extraction model from a result-driven perspective. This difference is the heterogeneous model performance loss. The embodiment of the present application believes that for the same sample sequence, it is normal for the first feature extraction model and the second feature extraction model to extract different information due to different focus positions, but the vertical distribution of information in the sample sequence should be consistent. Accordingly, the detection results should also be consistent. If they are inconsistent, heterogeneous model performance loss will occur.

[0094] Step 18: Obtain a first ship detection loss based on the difference between the first detection result distribution and the annotation information corresponding to the sample sequence.

[0095] The purpose of feature extraction is to obtain accurate detection results. Therefore, the parameters of the model related to feature extraction should be adjusted based on the detection results. That is to say, steps 18 and 19 characterize the loss from the perspective of the difference between the predicted value and the true value.

[0096] Step 19: Obtain a second ship detection loss based on the difference between the second detection result distribution and the annotation information corresponding to the sample sequence.

[0097] Step 20: Determine a model training loss based on the heterogeneous model ontology loss, the heterogeneous model performance loss, the first ship detection loss, and the second ship detection loss.

[0098] In the embodiment of the present application, the model training loss can be determined in a weighted manner. Of course, the weight value is not limited, and those skilled in the art can set it according to the actual situation, which will not be described in detail.

[0099] Step 21: Adjust the parameters of the first feature extraction model and the parameters of the second extraction model based on the model training loss.

[0100] The parameter adjustment method and training iteration are existing technologies and will not be elaborated on here. After the training is completed, a joint model is obtained based on the first feature extraction model obtained after adjusting the parameters and the ship detection model, and the joint model is used as the second object detection model. Through the above-mentioned training method, the first feature extraction model and the second feature extraction model have fully exchanged information with each other and fully absorbed each other's knowledge. The first feature extraction model itself can extract global feature information. Now, after training, the extraction capability of key positions has been increased, and the feature extraction capability has been significantly enhanced. The method for obtaining the ship detection model is not limited and can refer to the existing technology, but the second object detection model is undoubtedly a heavyweight model because it has a very strong information extraction capability and a relatively slow speed, but the extracted information content is very rich and the detection accuracy is also high.

[0101] Based on this, extracting vessel features from both the target frame and the monitoring results includes extracting vessel features from both the target frame and the monitoring results using the first feature extraction model in the second object detection model. Identifying a vessel based on the vessel feature extraction results includes inputting the vessel feature extraction results into the vessel detection model in the second object detection model for vessel identification. Of course, the record management module is used to generate monitoring records based on the vessel identification results and the vessel feature extraction results. The specific method for generating such records may utilize existing technologies and is not limited in this embodiment of the present application.

[0102] The embodiment of the present application provides a reservoir-oriented video monitoring and dam safety monitoring system. By arranging a camera in a target area where a reservoir and a dam are located, automatic shooting can be performed on each position in the target area which needs to be automatically monitored. By analyzing video data obtained by shooting, any object which does not belong to the target area and illegally enters the target area can be automatically found, and different alarm strategies are automatically triggered according to different types of objects. The embodiment of the present application effectively decomposes the supervision task of the target area by designing multiple models with different emphases, thereby solving the real-time problem, that is, the embodiment of the present application can realize high-intensity supervision in real time. Specifically, for objects which are not very important and illegally enter the target area, real-time discovery and real-time alarm can be realized, and for objects which are very important and illegally enter the target area, real-time discovery, real-time alarm, real-time pursuit, object detailed information mining and recording can be realized. Therefore, scientific supervision is completed in different levels and different situations. However, it should be noted that the present application can realize real-time control regardless of the situation, which is due to the advantages of the model design and system architecture design in the embodiment of the present application. By realizing functional decoupling and uniform distribution of loads, each component of the system can perform its own function, and the load is balanced, the running performance of the components is balanced, and there is no case of slowing down the overall system response speed. This is the key to realizing real-time control in the embodiment of the present application.

[0103] The above merely describes preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. Video surveillance and dam safety monitoring system for reservoirs, characterized by: The system includes a camera cluster, a monitoring module, an alarm module, and a record management module. The camera cluster includes multiple cameras, and the video data captured by each camera forms a full field of view covering the target area where the reservoir and the dam are located; The monitoring module is configured to acquire at least one channel of video data, any of which is captured by a corresponding camera in the camera cluster, the corresponding camera being configured to continuously capture the reservoir or dam; and, for any of the video data, in response to the presence of a target frame in which a target object is detected in the video data, perform persistent object detection on the target object based on the video data by fusing gradient-based optical flow and image features, and feed back the target frame and monitoring results to the alarm module; The alarm module is used to extract ship features from the target frame and the monitoring result based on the second object detection model, identify the ship based on the ship feature extraction result, and issue an alarm according to the ship identification result; and, transmitting the vessel identification result and the vessel feature extraction result to the record management module; The record management module is used to generate a monitoring record based on the vessel identification result and the vessel feature extraction result; The second object detection model is trained by the following method: obtaining a set of sample images of ships, wherein each sample sequence in the set of sample images of ships includes multiple sample images of ships and corresponding annotation information, and each of the multiple sample images of ships includes the same ship; for any sample sequence, independently inputting each sample image in the sample sequence into a first feature extraction model and a second feature extraction model, thereby correspondingly obtaining a first sample information matrix sequence and a second sample information matrix sequence, respectively, wherein the first feature extraction model is a pre-trained model for extracting overall features of a ship, and the second feature extraction model is a pre-trained model for extracting features of key areas of a ship; Downsampling each first sample information matrix in the first sample information matrix sequence to obtain a first sample information distribution; Downsampling each second sample information matrix in the second sample information matrix sequence to obtain a second sample information distribution; Inputting the first sample information matrix sequence into a pre-trained ship detection model to obtain a first detection result distribution; inputting the second sample information matrix sequence into the pre-trained ship detection model to obtain a second detection result distribution; Obtaining a heterogeneous model ontology loss based on a difference between the first sample information distribution and the second sample information distribution; obtaining a heterogeneous model performance loss based on a difference between the first detection result distribution and the second detection result distribution; obtaining a first ship detection loss based on a difference between the first detection result distribution and the labeled information corresponding to the sample sequence; obtaining a second ship detection loss based on a difference between the second detection result distribution and the labeled information corresponding to the sample sequence; and determining a model training loss based on the heterogeneous model ontology loss, the heterogeneous model performance loss, the first ship detection loss, and the second ship detection loss. Parameters of the first feature extraction model and parameters of the second feature extraction model are adjusted based on the model training loss.

2. The reservoir-oriented video surveillance and dam safety monitoring system according to claim 1 is characterized in that: The target object is an object identified as a vessel, the target frame is a frame in which the target object is first detected in the video data within a first time interval, and the monitoring module is configured to perform the following operations: Sequentially extracting each initial image frame from the video data within the first time interval; The following first operation is performed for each initial image frame: Performing channel-wise connection processing on the initial image frame and a preset reference frame to obtain a target image frame, where the preset reference frame is an image frame captured by the camera corresponding to the video data without capturing any objects that do not belong to the target area, or a fusion result thereof, and the preset reference frame has the same clarity, field of view, and number of channels as the initial image frame; and detecting the target image frame based on a first object detection model; wherein the first object detection model is a lightweight object detection model for detecting illegally entered objects, the first object detection model is trained to classify object types, and the classification accuracy of the first object detection model is lower than a first threshold and the running speed is higher than a second threshold; In response to the first detection of a target object, the initial image frame where the target object is located is determined as the target frame, and the first operation is stopped.

3. The reservoir-oriented video surveillance and dam safety monitoring system according to claim 2 is characterized in that: The monitoring module is also used to perform the following operations: During the execution of the first operation, in response to detecting an illegal entry of an object other than the target object, transmitting an initial image frame containing the object to the alarm module, triggering an alarm of a prompt level of the alarm module; In response to the first detection of the target object, the target frame is fed back to the alarm module to trigger an alarm of an alarm level of the alarm module.

4. The reservoir-oriented video surveillance and dam safety monitoring system according to claim 3 is characterized in that: The performing persistent object detection on the target object based on the video data by fusing gradient-based optical flow and image features includes: For other image frames in the video data within the first time interval that are located after the initial image frame where the target object is located, the second operation is performed in sequence: The second operation is specifically: Performing gradient-based optical flow object mapping on all target objects in a frame previous to the current frame, and obtaining an object detection result of the current frame based on the mapping result, wherein the object detection result includes all target objects in the current frame; Obtaining a frame identifier of the current frame and a frame identifier of the target frame; In response to a situation where a remainder of the frame identifier of the target frame relative to the frame identifier of the current frame is zero, performing object detection based on image features on the current frame, and updating an object detection result of the current frame according to the detection result; In response to the situation that all target objects in the object detection result of the current frame are not completely the same as all target objects in the previous frame, the current frame is determined as the monitoring result.

5. The reservoir-oriented video surveillance and dam safety monitoring system according to claim 4 is characterized in that: The extracting of vessel features from the target frame and the monitoring result includes: The target frame and each image in the monitoring result are respectively input into a second object detection model for vessel feature extraction; wherein the second object detection model is a heavyweight information extraction model, the second object detection model is trained as a model for vessel feature extraction, and the extraction accuracy of the second object detection model is higher than a third threshold and the operating speed is lower than a fourth threshold.

6. The reservoir-oriented video surveillance and dam safety monitoring system according to claim 1, characterized in that: After the training is completed, a joint model is obtained based on the first feature extraction model obtained after adjusting the parameters and the ship detection model, and the joint model is used as the second object detection model.

7. The reservoir-oriented video surveillance and dam safety monitoring system according to claim 6 is characterized in that: The extracting ship features from both the target frame and the monitoring result includes: extracting ship features from both the target frame and the monitoring result using a first feature extraction model in the second object detection model; The performing ship identification based on the ship feature extraction result includes: inputting the ship feature extraction result into a ship detection model in the second object detection model to perform ship identification.

Images