Method and system for realizing automatic staring based on video analysis

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, in particular to a method and system for achieving automatic staring based on video analysis, and the method comprises the following steps: target marking detection, model training, video stream access, frame extraction, reasoning, and result confirmation and processing. The method has the beneficial effects that through target-based intelligent video analysis and combination of various segmentation, electronic fence, detection and tracking technologies, distinguishing and identification of various abnormal objects or wearing are solved, interference caused by bad conditions in a complex scene under an all-weather condition is eliminated, and the method is mainly applied to all-weather monitoring of a working field scene, and has a wide application prospect. And the monitoring is more accurate and timely. In addition, by adding a new training model and an alarm object and setting the alarm object and the alarm period, the alarm is more accurate.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a method and system for automatic surveillance based on video analysis. Background Technology

[0002] Traditional communication systems require continuous manual monitoring at construction sites to prevent non-operational personnel and vehicles from entering the site. When there are multiple work sites or long-term operations, a significant amount of manpower and time is required, and human negligence may lead to non-operational personnel and vehicles entering the work site.

[0003] While AI recognition rates are high under specific conditions, existing models still suffer from insufficient robustness and generalization ability in complex and variable real-world surveillance environments (such as extreme weather, drastic lighting changes, severe occlusion, low resolution, and camera shake), leading to false alarms, missed alarms, or decreased recognition performance. Furthermore, with the massive increase in surveillance data, efficiently storing, processing, and analyzing this data has become a major challenge. Existing technologies cannot quickly and accurately extract valuable clues from large amounts of data. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for automatic surveillance based on video analysis, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for automatic monitoring based on video analysis, comprising the following steps:

[0006] Target labeling: Collect target materials, determine the types of targets that need to be monitored, and collect relevant image and video materials from multiple channels to ensure the diversity and richness of the materials; use labeling tools to label the collected materials, and the labeling content includes the target location, category and possible attributes;

[0007] Model Training: Build a training library, integrate labeled historical materials and on-site implementation footage, and preprocess the data; select a suitable deep learning model architecture according to the requirements of the object detection task, and adjust and optimize the selected architecture; train the model using labeled training data, set a loss function to measure the difference between the prediction results and the true labels, adjust the model parameters through optimization algorithms, save the model weights regularly, evaluate the model performance using the validation set and adjust the training strategy according to the indicators, and test and evaluate the model after training is completed.

[0008] Preferably, in the target labeling step, when collecting target materials, if monitoring a specific vehicle in a security scenario, it is necessary to extract materials by taking pictures and video clips of the vehicle from different angles, lighting conditions, and background environments; the labeling tools include LabelImg and CVAT.

[0009] Preferably, the method further includes the following steps: Accessing the video stream: Obtaining the target video stream address by calling the API interface of the video cloud platform and inputting relevant parameters to obtain the access address; using a video access device or software that supports the GB28181 protocol to access the acquired video stream into the local system. GB28181 is a standard protocol for video surveillance networking.

[0010] Preferably, the method further includes the following steps: frame extraction: using a video processing library to read and extract frames from the incoming video stream in real time, and setting the frame extraction frequency according to the actual application scenario; extracting the image after frame extraction according to the electronic fence rules. The electronic fence rules can be manually set by setting the coordinate range or automatically generated by using an image segmentation algorithm. When using an image segmentation algorithm, the target area and background area are first segmented and identified in the image, and then the electronic fence is set according to the position and shape of the target area.

[0011] Preferably, the following steps are also included:

[0012] Inference: Image data extracted by frame extraction and electronic fence is pushed to the AI ​​platform via network communication. When sending, the image data is encoded and formatted, and necessary parameters such as model name and version number are transmitted. The AI ​​platform performs inference on the image based on the specified model, analyzes the image content, and identifies the target location and category information.

[0013] Result Confirmation and Processing: The AI ​​platform returns the inference results to the front-end system. The results include the target bounding box coordinates, category labels, and confidence information. The front-end system performs subsequent processing based on the results, matches the inference results with system rules, and reports to management personnel to trigger an alarm.

[0014] A system for an automatic monitoring method based on video analytics includes:

[0015] Target Annotation Module: Used to collect target materials, determine the types of targets that need to be monitored, collect image and video materials related to the targets from multiple channels to ensure the diversity and richness of the materials; use annotation tools to annotate the collected materials, and the annotation content includes the target location, category and possible attributes;

[0016] Model training module: Build a training library, integrate labeled historical materials and on-site implementation footage, and preprocess the data; select a suitable deep learning model architecture according to the requirements of the object detection task, and adjust and optimize the selected architecture; train the model using labeled training data, set a loss function to measure the difference between the prediction results and the true labels, adjust the model parameters through optimization algorithms, save the model weights periodically, evaluate the model performance using the validation set and adjust the training strategy according to the indicators, and test and evaluate the model after training is completed.

[0017] Preferably, in the target labeling module, when collecting target materials, if a specific vehicle is being monitored in a security scenario, images and video clips of the vehicle are taken from different angles, lighting conditions, and background environments to extract materials. The sources of materials include historical monitoring records, public datasets, and on-site shooting; the labeling tools include LabelImg and CVAT.

[0018] Preferably, it also includes: a video stream access module: obtaining the target video stream address by calling the API interface of the video cloud platform, and obtaining the access address by inputting relevant parameters; using a video access device or software that supports the GB28181 protocol to access the obtained video stream to the local system, where GB28181 is the standard protocol for video surveillance networking.

[0019] Preferably, it also includes: a frame extraction module: using a video processing library to perform real-time reading and frame extraction operations on the incoming video stream, and setting the frame extraction frequency according to the actual application scenario; extracting the image after frame extraction according to the electronic fence rules, which can be manually set by setting the coordinate range or automatically generated by using an image segmentation algorithm. When using an image segmentation algorithm, the target area and background area are first segmented and identified in the image, and then the electronic fence is set according to the position and shape of the target area.

[0020] Preferred options also include:

[0021] Inference module: Pushes image data extracted from frames and captured by electronic fences to the AI ​​platform via network communication. When sending, the image data is encoded and formatted, and necessary parameters such as model name and version number are transmitted. The AI ​​platform performs inference on the image based on the specified model, analyzes the image content, and identifies the target location and category information.

[0022] Result Confirmation and Processing Module: Receives the inference results returned by the AI ​​platform, including target bounding box coordinates, category labels, and confidence information; performs subsequent processing based on the results, matches the inference results with system rules, and reports to management personnel to trigger an alarm.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] This invention proposes a method and system for automatic monitoring based on video analytics. Through target-based intelligent video analysis, combined with various segmentation, geofencing, detection, and tracking technologies, it solves the problem of distinguishing and identifying various abnormal objects or clothing, eliminating interference caused by adverse conditions in all-weather and complex scenarios. It is primarily applied to all-weather monitoring in work sites, making monitoring more accurate and timely. Furthermore, by incorporating new training models and alarm targets, as well as setting alarm targets and alarm cycles, the alarm accuracy is enhanced. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1: This invention provides a technical solution: a method for automatic monitoring based on video analysis, comprising the following steps:

[0028] a. Target Annotation. First, target material collection: determine the type of target to be monitored, such as specific personnel or vehicles in a security scenario. Collect a large amount of image and video material related to the target from various scenarios. For example, if the target is to monitor a specific type of vehicle, material can be extracted from vehicle images and video clips taken from different angles, under different lighting conditions, and in different background environments. This material can come from various sources such as historical surveillance records, public datasets, and on-site shooting. Ensure the diversity and richness of the material so that the model can learn the characteristics of the target under various conditions. Second, target annotation: use annotation tools (such as LabelImg, CVAT, etc.) to annotate the collected material. The annotation content includes the target's location (usually marked with a bounding box to indicate the target's outline), category (such as vehicle, personnel, etc.), and possible attributes (such as vehicle color, personnel gender, etc.).

[0029] b. Model Training. First, a training library is built, integrating labeled historical footage and on-site implementation footage (after annotation processing) into a single model training library. On-site implementation footage refers to the latest video or image footage acquired in real-world application scenarios, reflecting changes in target features under the current environment. The data in the training library is preprocessed, including data augmentation (such as random cropping, rotation, and color adjustment) and normalization, to increase data diversity and the model's generalization ability. For example, random cropping can simulate the appearance of targets in different sizes of the field of view, and color adjustment can enhance the model's adaptability to different lighting conditions. Second, a model architecture is selected based on the requirements of the object detection task. Common object detection model architectures include YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and Faster R-CNN. Each architecture has its advantages and disadvantages; for example, the YOLO series of models has fast detection speed and is suitable for real-time applications; Faster R-CNN has higher accuracy but is relatively slower.

[0030] The selected model architecture is appropriately adjusted and optimized to suit the characteristics of the current task. This includes adjusting the model's input size, modifying the number of layers or channels in the network, etc. Next, during training, the model is trained using labeled training data. During training, the model learns how to identify the location and category of targets from input image or video frames. A loss function (such as cross-entropy loss, Smooth L1 loss, etc.) is set to measure the difference between the model's predictions and the ground truth labels, and optimization algorithms (such as stochastic gradient descent SGD, Adam, etc.) are used to adjust the model's parameters, gradually reducing the value of the loss function. Finally, during training, the model's weights are periodically saved, and its performance is evaluated using a validation set. Training strategies are adjusted based on the validation set performance metrics (such as accuracy, recall, MAP, etc.), such as adjusting the learning rate or increasing the training data. After training, the model is tested and evaluated to ensure it can accurately detect targets in real-world application scenarios.

[0031] c. Access the video stream. Obtain the address of the target video stream by calling the API interface of the video cloud platform. The video cloud platform usually provides a video stream URL, which points to the video resource stored in the cloud. For example, using the API interface provided by the video cloud platform, inputting relevant parameters (such as the video stream ID, access permissions, etc.), you can obtain the access address of a video stream, such as "rtsp: / / xxx.com / streaming / channels / 1".

[0032] Video streams are accessed via GB28181, a video surveillance networking standard protocol used to enable interconnection between video surveillance devices. Video access devices or software that support the GB28181 protocol are used to connect the acquired video streams to the local system.

[0033] d. Frame Extraction. Real-time processing of the incoming video stream. Since the video stream is a continuous sequence of images, frame extraction is necessary to reduce computation and improve processing speed. The frame extraction frequency is set according to the requirements of the actual application scenario. A video processing library (such as OpenCV) is used to read the video stream and perform frame extraction. OpenCV provides rich video processing functions, and the frame extraction interval can be controlled by setting parameters. Capturing according to electronic fence rules: The extracted image is cropped according to pre-set electronic fence rules. Electronic fence rules can be manually set with coordinate ranges (such as the coordinates of the top left and bottom right corners) or automatically generated by an image segmentation algorithm. If an image segmentation algorithm is used, the image can be segmented first to identify the target and background regions, and then an electronic fence can be set based on the position and shape of the target region.

[0034] e. Inference. The captured image data, after frame extraction and geofencing, is pushed to the AI ​​platform for inference. Image data is sent to the AI ​​platform via network communication (such as HTTP, TCP / IP, etc.). During transmission, the image data needs to be appropriately encoded and formatted to ensure correct transmission to the AI ​​platform. Simultaneously, necessary parameters, such as the model name and version number, need to be passed to the AI ​​platform so that it can correctly call the corresponding model for inference. After receiving the image data, the AI ​​platform performs inference on the image based on the specified model. The model analyzes the content in the image, identifying the location, category, and other information of the target.

[0035] f. Result confirmation and processing.

[0036] After inference is complete, the AI ​​platform returns the inference results to the front-end system. The inference results typically include information such as the target's bounding box coordinates, category label, and confidence score. Upon receiving the inference results, the front-end system can perform further processing, matching the results against system rules and reporting to administrators to trigger alarms.

[0037] Example 2, based on Example 1, proposes a system for an automatic monitoring method based on video analysis, comprising:

[0038] The target detection and annotation module is used to collect target materials, determine the types of targets to be monitored, and collect image and video materials related to the targets from multiple channels to ensure the diversity and richness of the materials. The collected materials are annotated using annotation tools, and the annotation content includes the target location, category and possible attributes. When collecting target materials, if monitoring a specific vehicle in a security scenario, the vehicle images and video clips are taken from different angles, lighting conditions and background environments to extract materials. The material sources include historical monitoring records, public datasets and on-site shooting. The annotation tools include LabelImg and CVAT.

[0039] Model training module: Build a training library, integrate labeled historical materials and on-site implementation footage, and preprocess the data; select a suitable deep learning model architecture according to the requirements of the object detection task, and adjust and optimize the selected architecture; train the model using labeled training data, set a loss function to measure the difference between the prediction results and the true labels, adjust the model parameters through optimization algorithms, save the model weights periodically, evaluate the model performance using the validation set and adjust the training strategy according to the indicators, and test and evaluate the model after training is completed.

[0040] Video stream access module: Obtain the target video stream address by calling the API interface of the video cloud platform, and obtain the access address by inputting relevant parameters; use video access devices or software that support the GB28181 protocol to access the acquired video stream to the local system. GB28181 is the standard protocol for video surveillance networking.

[0041] Frame extraction module: Uses a video processing library to read and extract frames from the incoming video stream in real time, and sets the frame extraction frequency according to the actual application scenario; extracts the image after frame extraction according to the electronic fence rules. The electronic fence rules can be manually set by setting the coordinate range or automatically generated by using an image segmentation algorithm. When using an image segmentation algorithm, the target area and background area are first segmented and identified in the image, and then the electronic fence is set according to the position and shape of the target area.

[0042] Inference module: Pushes image data extracted from frames and captured by electronic fences to the AI ​​platform via network communication. When sending, the image data is encoded and formatted, and necessary parameters such as model name and version number are transmitted. The AI ​​platform performs inference on the image based on the specified model, analyzes the image content, and identifies the target location and category information.

[0043] Result Confirmation and Processing Module: Receives the inference results returned by the AI ​​platform, including target bounding box coordinates, category labels, and confidence information; performs subsequent processing based on the results, matches the inference results with system rules, and reports to management personnel to trigger an alarm.

[0044] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatic monitoring based on video analytics, characterized in that: Includes the following steps: Target labeling: Collect target materials, determine the types of targets that need to be monitored, and collect relevant image and video materials from multiple channels to ensure the diversity and richness of the materials; use labeling tools to label the collected materials, and the labeling content includes the target location, category and possible attributes; Model Training: Build a training library, integrate labeled historical materials and on-site implementation footage, and preprocess the data; select a suitable deep learning model architecture according to the requirements of the object detection task, and adjust and optimize the selected architecture; train the model using labeled training data, set a loss function to measure the difference between the prediction results and the true labels, adjust the model parameters through optimization algorithms, save the model weights regularly, evaluate the model performance using the validation set and adjust the training strategy according to the indicators, and test and evaluate the model after training is completed.

2. The method for automatic monitoring based on video analysis according to claim 1, characterized in that: In the target labeling step, when collecting target materials, if monitoring a specific vehicle in a security scenario, it is necessary to extract materials by taking pictures and video clips of the vehicle from different angles, lighting conditions, and background environments; the labeling tools include LabelImg and CVAT.

3. The method for automatic monitoring based on video analysis according to claim 2, characterized in that: It also includes the following steps: Accessing the video stream: Obtain the target video stream address by calling the API interface of the video cloud platform and inputting relevant parameters to obtain the access address; use video access devices or software that support the GB28181 protocol to access the acquired video stream to the local system. GB28181 is the standard protocol for video surveillance networking.

4. The method for automatic monitoring based on video analysis according to claim 3, characterized in that: It also includes the following steps: frame extraction: use a video processing library to read and extract frames from the incoming video stream in real time, and set the frame extraction frequency according to the actual application scenario; capture the extracted image according to the electronic fence rules. The electronic fence rules can be manually set by setting the coordinate range or automatically generated by using an image segmentation algorithm. When using an image segmentation algorithm, first segment and identify the target area and background area of ​​the image, and then set the electronic fence according to the position and shape of the target area.

5. The method for automatic monitoring based on video analysis according to claim 4, characterized in that: It also includes the following steps: Inference: Image data extracted by frame extraction and electronic fence is pushed to the AI ​​platform via network communication. When sending, the image data is encoded and formatted, and necessary parameters such as model name and version number are transmitted. The AI ​​platform performs inference on the image based on the specified model, analyzes the image content, and identifies the target location and category information. Result Confirmation and Processing: The AI ​​platform returns the inference results to the front-end system. The results include the target bounding box coordinates, category labels, and confidence information. The front-end system performs subsequent processing based on the results, matches the inference results with system rules, and reports to management personnel to trigger an alarm.

6. A system for the automatic surveillance method based on video analysis as described in claim 5, characterized in that: include: Target Annotation Module: Used to collect target materials, determine the types of targets that need to be monitored, collect image and video materials related to the targets from multiple channels to ensure the diversity and richness of the materials; use annotation tools to annotate the collected materials, and the annotation content includes the target location, category and possible attributes; Model training module: Build a training library, integrate labeled historical materials and on-site implementation footage, and preprocess the data; select a suitable deep learning model architecture according to the requirements of the object detection task, and adjust and optimize the selected architecture; train the model using labeled training data, set a loss function to measure the difference between the prediction results and the true labels, adjust the model parameters through optimization algorithms, save the model weights periodically, evaluate the model performance using the validation set and adjust the training strategy according to the indicators, and test and evaluate the model after training is completed.

7. The system according to claim 6, characterized in that: In the target labeling module, when collecting target materials, if a specific vehicle is being monitored in a security scenario, images and video clips of the vehicle are taken from different angles, lighting conditions, and background environments to extract materials. The sources of materials include historical monitoring records, public datasets, and on-site shooting; the labeling tools include LabelImg and CVAT.

8. The system according to claim 7, characterized in that: Also includes: Video stream access module: Obtain the target video stream address by calling the API interface of the video cloud platform, and obtain the access address by inputting relevant parameters; use video access devices or software that support the GB28181 protocol to access the acquired video stream to the local system. GB28181 is the standard protocol for video surveillance networking.

9. A system according to claim 8, characterized in that: Also includes: Frame extraction module: Uses a video processing library to read and extract frames from the incoming video stream in real time, and sets the frame extraction frequency according to the actual application scenario; extracts the image after frame extraction according to the electronic fence rules. The electronic fence rules can be manually set by setting the coordinate range or automatically generated by using an image segmentation algorithm. When using an image segmentation algorithm, the target area and background area are first segmented and identified in the image, and then the electronic fence is set according to the position and shape of the target area.

10. A system according to claim 9, characterized in that: Also includes: Inference module: Pushes image data extracted from frames and captured by electronic fences to the AI ​​platform via network communication. When sending, the image data is encoded and formatted, and necessary parameters such as model name and version number are transmitted. The AI ​​platform performs inference on the image based on the specified model, analyzes the image content, and identifies the target location and category information. Result Confirmation and Processing Module: Receives the inference results returned by the AI ​​platform, including target bounding box coordinates, category labels, and confidence information; performs subsequent processing based on the results, matches the inference results with system rules, and reports to management personnel to trigger an alarm.

Images