Water station anti-human interference early warning method based on lightweight pre-trained visual large model

Abstract

The present invention belongs to the field of target detection technology, and relates to a water station anti-human interference early warning method based on a lightweight pre-trained visual large model. The method comprises: pre-training the visual large model using a target tracking data set; training the pre-trained large model using a self-made data set to obtain a trained visual large model; jointly training the classification network and the segmentation network using a knowledge distillation transfer learning method; segmenting the image area through a lightweight visual large model; issuing an early warning for targets close to the early warning area based on the prediction results and issuing an early warning based on the residence time of targets not close to the early warning area; predicting the target behavior based on the items carried by the identified target and issuing an early warning based on the target behavior. The present invention can reduce the probability of false detection and false alarm; improve the accuracy of management by monitoring and warning the behavior of fast-moving targets; and enable the system to have the ability to migrate to different scenarios and targeted learning capabilities.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular, relates to a water station anti-human interference early warning method based on a lightweight pre-trained visual large model. Background Art

[0002] At present, manual patrol is often used in water station river management. The management of dangerous behaviors of personnel is often carried out through a combination of manual remote monitoring and on-site patrol. Although this method can achieve certain results, it consumes a lot of manpower and material resources for monitoring and patrolling, has low management efficiency, and cannot predict dangerous behaviors, thus causing danger.

[0003] At present, pedestrian detection in water stations often uses simple target detection methods, which usually directly detects whether there are people in the image, but does not effectively distinguish dangerous areas. The monitoring area is large, which increases the probability of false alarms and increases the management workload; the dangerous behavior judgment method based on target detection judges the pedestrian's stay time by detecting multiple images, but also monitors and warns pedestrians passing by, which often leads to false alarms and invalid monitoring behaviors; the dangerous behavior judgment method of water stations based on target detection uses visual detection models for detection, but such situations often have problems such as low model accuracy and inability to achieve migration and update; the dangerous behavior judgment method of water stations based on target detection uses visual detection models to detect pedestrians to determine whether there are dangerous behaviors, and there are cases of false alarms for pedestrians passing by and non-dangerous behaviors. Therefore, the accuracy of the existing water station anti-task interference warning method is low. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a water station anti-human interference early warning method based on a lightweight pre-trained visual large model, comprising:

[0005] Build a large Transformer-based visual model;

[0006] The target tracking data set is used to pre-train the visual large model to obtain a pre-trained large model;

[0007] Use annotation tools to annotate the dataset with detection targets and the water surface area segmentation map to obtain a self-made dataset;

[0008] Use the self-made data set to train the pre-trained large model to obtain the trained visual large model;

[0009] The trained visual model is compressed by pruning, and the knowledge distillation transfer learning method is used to fix the backbone network parameters of the visual model. The classification network and segmentation network are jointly trained to obtain a lightweight visual model.

[0010] The image is segmented by a lightweight visual large model to obtain dangerous areas in the water, warning areas close to the shore, and safe areas far from the shore;

[0011] Track the targets close to the warning area, detect the target's running trajectory, predict the trajectory trend of the targets close to the warning area based on their running trajectory and movement speed, issue warnings to the targets close to the warning area based on the prediction results, and issue warnings based on the residence time of the targets not close to the warning area;

[0012] Identify the objects carried by the target close to the warning area, predict the target behavior based on the identified objects carried by the target, and issue a warning based on the predicted target behavior;

[0013] Alarm for targets entering dangerous areas.

[0014] Based on the above technical solution, the present invention can also be improved as follows.

[0015] Furthermore, the large visual model is a Transformer-based convolutional neural network model built using pytorch; the target tracking dataset is a public dataset.

[0016] Furthermore, Labelme is used to label the segmented areas, and Labelmg is used to label the targets.

[0017] Furthermore, based on the running trajectory and movement speed of the target close to the warning area, the trajectory trend of the target close to the warning area is predicted, including: calculating the positions of multiple future time nodes of the target based on the running trajectory, movement direction and movement speed of the target, and obtaining the trajectory trend of the target close to the warning area.

[0018] Furthermore, the trajectory trend of the target is simulated and displayed at the current moment according to the positions of multiple future time nodes of the target.

[0019] Furthermore, the items carried by the target close to the warning area are identified, including:

[0020] Construct an item sample data set; the item sample data set includes item feature data and item names;

[0021] Construct a neural network model and preset the parameters of the neural network model;

[0022] Taking the item feature data as input and the item name as output, the neural network model is trained using the item sample data set to obtain an item recognition neural network model;

[0023] The item feature data of the items carried by the target close to the warning area is extracted and input into the item recognition neural network model to obtain the item name.

[0024] Further, the target behavior is predicted based on the identified items carried by the target, including: constructing a database of item names and behaviors corresponding to the item names; matching the behaviors corresponding to the item names according to the item names to obtain the target behavior prediction result.

[0025] Further, an early warning is issued according to the residence time of the target not close to the early warning area, including: judging the residence time of the target not close to the early warning area, if the residence time is greater than a set threshold, an early warning is issued, otherwise the running trajectory of the target is continuously detected.

[0026] The beneficial effects of the present invention are:

[0027] (1) The pedestrian detection method currently widely used in water station monitoring and management often adopts a simple target detection method without a dangerous area division method. The present invention adopts a visual large model detection method to divide the dangerous area, detect the dangerous area and target in the image, and divide the target categories in different areas to more accurately predict the dangerous behavior of the target, thereby reducing the probability of false detection and false alarm;

[0028] (2) In contrast to the currently commonly used water station monitoring method based on target detection, which simply detects pedestrians through multiple images to obtain the dwell time, the present invention predicts the movement trajectory and continuous dwell time of pedestrians and vehicles by using a video stream method to predict dangerous behaviors, and improves the accuracy of management by monitoring and warning the behaviors of fast-moving targets;

[0029] (3) In view of the fact that data sets are difficult to obtain in the current water station monitoring method through target detection, resulting in low target detection accuracy, the present invention adopts a lightweight pre-trained visual big model. The visual big model can be continuously trained through data collected from different sites, thereby improving the performance of the monitoring system, and enabling the system to have the ability to migrate to different scenes and targeted learning capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The schematic diagram of the water station anti-human interference early warning method based on the lightweight pre-trained visual large model provided by the present invention;

[0031] Figure 2 This is a schematic diagram of the principle of a lightweight visual macro model;

[0032] Figure 3 The present invention is a flowchart of a specific implementation method of a water station anti-human interference early warning method based on a lightweight pre-trained visual large model. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0034] As an example, Figure 1 and attached Figure 2 As shown, in order to solve the above technical problems, this embodiment provides a water station anti-human interference early warning method based on a lightweight pre-trained visual large model, including:

[0035] Build a large Transformer-based visual model;

[0036] The target tracking data set is used to pre-train the visual large model to obtain a pre-trained large model;

[0037] Use annotation tools to annotate the dataset with detection targets and the water surface area segmentation map to obtain a self-made dataset;

[0038] Use the self-made data set to train the pre-trained large model to obtain the trained visual large model;

[0039] The trained visual model is compressed by pruning, and the knowledge distillation transfer learning method is used to fix the backbone network parameters of the visual model. The classification network and segmentation network are jointly trained to obtain a lightweight visual model.

[0040] The image is segmented by a lightweight visual large model to obtain dangerous areas in the water, warning areas close to the shore, and safe areas far from the shore;

[0041] Track the targets close to the warning area, detect the target's running trajectory, predict the trajectory trend of the targets close to the warning area based on their running trajectory and movement speed, issue warnings to the targets close to the warning area based on the prediction results, and issue warnings based on the residence time of the targets not close to the warning area;

[0042] Identify the objects carried by the target close to the warning area, predict the target behavior based on the identified objects carried by the target, and issue a warning based on the predicted target behavior;

[0043] Alarm for targets entering dangerous areas.

[0044] In actual application, the trained large visual model is compressed through pruning operations, and the transfer learning method of knowledge distillation is adopted to fix the backbone network parameters of the large visual model. The classification network and the segmentation network are jointly trained to achieve enhanced learning of target cognition in the scene, thereby achieving the optimal monitoring effect of the application scenario.

[0045] Optionally, the large visual model is a Transformer-based convolutional neural network model built using pytorch; the target tracking dataset is a public dataset.

[0046] The present invention uses pytorch to build a Transformer-based visual big model, and then uses public data sets and self-made data sets to pre-train the big model to obtain a pre-trained big model. Then, the obtained visual big model is compressed by pruning operation to obtain a lightweight visual big model, which can reduce the model's demand for computing power.

[0047] Optionally, use Labelme to label the segmented regions and use Labelmg to label the targets.

[0048] In actual application, for the segmented area annotation part, dangerous areas such as lakes and rivers are marked in red, dangerous areas within a set range close to the shore are marked in yellow, and safe areas far away from dangerous areas are marked in green. For the target annotation part, the annotation method in the COCO dataset is used to distinguish the target object from the environment.

[0049] Optionally, the trajectory trend of the target close to the warning area is predicted based on the running trajectory and movement speed of the target close to the warning area, including: calculating the positions of multiple future time nodes of the target based on the running trajectory, movement direction and movement speed of the target, to obtain the trajectory trend of the target close to the warning area.

[0050] In the actual application process, the lightweight visual large model is first used to segment the dangerous area of ​​the water, the warning area close to the shore, and the safe area far from the shore. Then different monitoring strategies are used to classify the targets in the three areas, as shown in the attached figure. Figure 3 As shown, the details are as follows:

[0051] (1) For the safe area, since this area is far away from the shore, vehicles and pedestrians maintain a certain safe distance from the dangerous area. The system calculates the position of multiple future time nodes of the target based on the running trajectory and movement speed of the vehicles and pedestrians, so as to determine whether the target is rapidly approaching the warning area. For targets that are not rapidly approaching the warning area, they are judged as safe targets and continuously tracked and detected, and warning judgments are made for targets that are rapidly approaching the warning area. Optionally, for vehicles, it is directly determined whether the target is rapidly approaching the dangerous area, and alarm processing is directly performed for targets that are rapidly approaching. For targets that are not rapidly approaching the dangerous area, it is determined whether they have stayed for more than 3 minutes. If so, an alarm is issued. For vehicles that pass quickly without stopping, continuous detection is performed without any warning processing;

[0052] (2) For pedestrians, the movement trajectory of pedestrians is monitored, and the items carried by pedestrians are monitored and identified. The intention of the pedestrians’ movements is predicted through the items in their hands, so as to achieve the ability to make advance predictions. For pedestrians holding bucket-shaped containers, a direct warning is issued to remind the back-end management staff to pay attention to the possibility of dangerous placement of objects. For pedestrians holding other objects, it is determined whether they are staying. For those who do not stay, continuous monitoring is performed without any treatment. For those who stay for less than 3 minutes, a warning is issued, and for those who stay for more than 3 minutes, an alarm is issued;

[0053] (3) For dangerous areas, any target object that stays in the dangerous area for a short period of time, less than 1 minute, needs to be alarmed to remind management personnel to deal with it in time.

[0054] Taking the monitoring of surface vessels and pedestrians as an example, for fast-moving vessels that do not stop, they are detected without warning. For pedestrian monitoring passing through the warning area, only a warning is issued if the pedestrian stays for less than 3 minutes. When a pedestrian is in the warning area and stays for more than 3 minutes, an alarm is issued.

[0055] Optionally, the trajectory trend of the target is simulated and displayed at the current moment based on the positions of multiple future time nodes of the target.

[0056] Optionally, identify items carried by targets approaching the warning area, including:

[0057] Construct an item sample data set; the item sample data set includes item feature data and item names;

[0058] Construct a neural network model and preset the parameters of the neural network model;

[0059] Taking the item feature data as input and the item name as output, the neural network model is trained using the item sample data set to obtain an item recognition neural network model;

[0060] The item feature data of the items carried by the target close to the warning area is extracted and input into the item recognition neural network model to obtain the item name.

[0061] Optionally, predicting the target behavior based on the identified items carried by the target includes: constructing a database of item names and behaviors corresponding to the item names; matching the behaviors corresponding to the item names according to the item names to obtain a target behavior prediction result.

[0062] Optionally, an early warning is issued based on the residence time of the target not close to the early warning area, including: determining the residence time of the target not close to the early warning area, and issuing an early warning if the residence time is greater than a set threshold, otherwise continuously detecting the target's running trajectory.

[0063] The present invention has the following advantages:

[0064] (1) The pedestrian detection method currently widely used in water station monitoring and management often adopts a simple target detection method without a dangerous area division method. The present invention adopts a visual large model detection method to divide the dangerous area, detect the dangerous area and target in the image, and divide the target categories in different areas to more accurately predict the dangerous behavior of the target, thereby reducing the probability of false detection and false alarm;

[0065] (2) In contrast to the currently commonly used water station monitoring method based on target detection, which simply detects pedestrians through multiple images to obtain the dwell time, the present invention predicts the movement trajectory and continuous dwell time of pedestrians and vehicles by using a video stream method to predict dangerous behaviors, and improves the accuracy of management by monitoring and warning the behaviors of fast-moving targets;

[0066] (3) In view of the fact that data sets are difficult to obtain in the current water station monitoring method through target detection, resulting in low target detection accuracy, the present invention adopts a lightweight pre-trained visual big model. The visual big model can be continuously trained through data collected from different sites, thereby improving the performance of the monitoring system, and enabling the system to have the ability to migrate to different scenes and targeted learning capabilities.

[0067] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A water station anti-human interference early warning method based on a lightweight pre-trained visual large model, characterized in that: include: Build a large Transformer-based visual model; The target tracking data set is used to pre-train the visual large model to obtain a pre-trained large model; Use annotation tools to annotate the dataset with detection targets and the water surface area segmentation map to obtain a self-made dataset; Use the self-made data set to train the pre-trained large model to obtain the trained visual large model; The trained visual model is compressed by pruning, and the knowledge distillation transfer learning method is used to fix the backbone network parameters of the visual model. The classification network and segmentation network are jointly trained to obtain a lightweight visual model. The image is segmented by a lightweight visual large model to obtain dangerous areas in the water, warning areas close to the shore, and safe areas far from the shore; Track the targets close to the warning area, detect the target's running trajectory, predict the trajectory trend of the targets close to the warning area based on their running trajectory and movement speed, issue warnings to the targets close to the warning area based on the prediction results, and issue warnings based on the residence time of the targets not close to the warning area; For safe areas, the system calculates the position of multiple future time nodes of the target based on the running trajectories and movement speeds of vehicles and pedestrians, so as to determine whether the target is rapidly approaching the warning area. Targets that are not rapidly approaching the warning area are judged as safe targets and continuously tracked and detected, and targets that are rapidly approaching the warning area are given a warning judgment. For dangerous areas, any target object that stays in the dangerous area for a short period of time within 1 minute will be alarmed to remind management personnel to handle it in time. Identify the objects carried by the target close to the warning area, predict the target behavior based on the identified objects carried by the target, and issue a warning based on the predicted target behavior; Alarm for targets entering dangerous areas.

2. According to claim 1, the water station anti-human interference early warning method based on lightweight pre-trained visual large model is characterized in that: The visual model is a Transformer-based convolutional neural network model built using pytorch; the target tracking dataset is a public dataset; the target tracking dataset is a public dataset.

3. According to claim 1, the water station anti-human interference early warning method based on lightweight pre-trained visual large model is characterized in that: Labelme is used to label the segmented area, and Labelmg is used to label the target.

4. According to claim 1, the water station anti-human interference early warning method based on lightweight pre-trained visual large model is characterized in that: According to the running trajectory and movement speed of the target close to the warning area, the trajectory trend of the target close to the warning area is predicted, including: according to the running trajectory, movement direction and movement speed of the target, the position of multiple future time nodes of the target is calculated to obtain the trajectory trend of the target close to the warning area.

5. According to claim 4, the water station anti-human interference early warning method based on lightweight pre-trained visual large model is characterized in that: The trajectory trend of the target is simulated and displayed at the current moment based on the positions of multiple future time nodes of the target.

6. According to claim 1, the water station anti-human interference early warning method based on lightweight pre-trained visual large model is characterized in that: Identify items carried by targets approaching the warning area, including: Construct an item sample data set; the item sample data set includes item feature data and item names; Construct a neural network model and preset the parameters of the neural network model; Taking the item feature data as input and the item name as output, the neural network model is trained using the item sample data set to obtain an item recognition neural network model; The item feature data of the items carried by the target close to the warning area is extracted and input into the item recognition neural network model to obtain the item name.

7. According to claim 1, the water station anti-human interference early warning method based on lightweight pre-trained visual large model is characterized in that: The target behavior is predicted according to the items carried by the identified target, including: constructing a database of item names and behaviors corresponding to the item names; matching the behaviors corresponding to the item names according to the item names to obtain the target behavior prediction result.

8. According to claim 1, the water station anti-human interference early warning method based on lightweight pre-trained visual large model is characterized in that: The warning is issued according to the residence time of the target not close to the warning area, including: judging the residence time of the target not close to the warning area, if the residence time is greater than the set threshold, issuing a warning, otherwise continuously detecting the running trajectory of the target.

Images