Deepstream-based monitoring video analysis method and system

Abstract

The application provides a video monitoring analysis method and system based on DeepStream, wherein the method comprises the following steps: installing DeepStream on a Jetson AGX Xavier embedded development board, wherein DeepStream is provided with a TensorRT-based inference plug-in for converting model files; deploying a trained lightweight target detection model into DeepSteam; inputting at least one camera-based multi-channel video stream transmitted based on an RTSP protocol into a video stream acquisition plug-in of DeepStream, and performing target detection on the multi-channel video stream in a pipeline flow mode by DeepStream; and encoding and compressing the processed video stream, and pushing the video stream to a result display module in a wireless network mode for display. The application quickly builds an intelligent video monitoring analysis system, and analyzes information in the multi-channel video stream in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video monitoring processing, and particularly relates to a monitoring video analysis method and system based on DeepStream. BACKGROUND

[0002] Intelligent video monitoring refers to tracking and detecting targets in video or monitoring by using video, digital image processing and analysis related technologies without human intervention and reacting to abnormal conditions.

[0003] Target detection in images and videos is a focus of computer vision research and a key technology in intelligent video monitoring systems. Current target detection technologies mainly rely on deep neural network feature extraction technology to identify the category of objects from pictures or videos while outputting the position coordinates of the targets. Video monitoring systems usually need to process data of multiple cameras, and often have requirements for the size of the device in specific scenarios, so the monitoring video system is usually deployed on an embedded device. However, the computing power of the embedded device is limited, and it is difficult to deploy a neural network model.

[0004] Therefore, how to deploy a target detection model on an embedded device while ensuring the running rate of the embedded device is a problem to be solved. SUMMARY

[0005] In view of the above problems existing in the related art, the embodiments of the present application provide a monitoring video analysis method and system based on DeepStream.

[0006] In a first aspect, the present application provides a monitoring video analysis method based on DeepStream, comprising:

[0007] installing DeepStream on a Jetson AGX Xavier embedded development board, wherein the DeepStream is provided with a TensorRT-based inference plug-in, and the TensorRT-based inference plug-in is used to convert a model file;

[0008] deploying a trained lightweight target detection model into the DeepSteam;

[0009] inputting multiple video streams transmitted by at least one camera based on an RTSP protocol into a video stream acquisition plug-in of the DeepStream, and performing target detection on the multiple video streams in a pipeline stream manner by the DeepStream;

[0010] encoding and compressing the processed video stream, and pushing the processed video stream to a result display module in a wireless network manner through an RTSP protocol for display.

[0011] In some embodiments, the lightweight target detection model is an improved VOLOv5s model, and the improved VOLOv5s model is obtained by the following steps:

[0012] The convolution module in the VOLOv5s backbone network is replaced with a depth separable convolution module, the Bottleneck module in the VOLOv5s neck network is replaced with a GhostBottleneck module, and channel pruning is performed on the replaced neck network.

[0013] In some embodiments, the channel pruning of the replaced neck network includes:

[0014] In the first model training process, based on a preset data set, the learning rate is periodically changed in a cosine annealing manner until the model converges to a preset expected position.

[0015] In the second model training process, the channels of the replaced neck network that have completed the first model training process are pruned according to a preset proportion, the data set and the hyperparameters are kept unchanged, fine-tuning training is performed, and the channel pruning is completed.

[0016] In some embodiments, the DeepStream performs target detection on the multi-channel video stream in a pipeline flow manner, including:

[0017] The video stream acquisition plug-in of the DeepStream transmits the received multi-channel video stream to the encoding and decoding plug-in of the DeepStream, and the encoding and decoding plug-in includes a first special-purpose chip for encoding and a second special-purpose chip for decoding.

[0018] The multi-channel video stream that has been hardware encoded by the encoding and decoding plug-in and buffered in the video memory is transmitted to the TensorRT-based inference plug-in, and the weight file of the lightweight target detection model is converted into a TensorRT-supported inference engine by the TensorRT-based inference plug-in.

[0019] Based on the TensorRT-supported inference engine, video structured information and target detection results of the multi-channel video stream are obtained.

[0020] The video structured information is taken as an inference result, and is output in text form through the first output end of the DeepStream.

[0021] Based on the position information of the target in the target detection result, a bounding box is drawn to the corresponding position of the video in the form of a bounding box, and multiple videos are spliced into one picture. After hardware coding of the coding and decoding plug-in and display memory buffering, the second output end of the DeepStream outputs.

[0022] In some embodiments, the target detection result of the multi-channel video stream is obtained based on the inference engine supported by the TensorRT, comprising:

[0023] Based on the inference engine supported by the TensorRT, the inference is performed frame by frame. When the lightweight target detection model detects a target in the current video frame, the category and position information of the target are output as the target detection result.

[0024] When the lightweight target detection model does not detect a target in the current video frame, the detection of the next video frame is performed.

[0025] In some embodiments, the CPU and GPU of the Jetson AGX Xavier embedded development board share a high-speed memory, and the display memory buffering process is performed in the high-speed memory.

[0026] In a second aspect, the embodiments of the present application also provide a monitoring video analysis system based on DeepStream, comprising:

[0027] At least one camera for collecting multi-channel video streams and transmitting the multi-channel video streams based on the RTSP protocol to an embedded device;

[0028] An embedded device using a Jetson AGX Xavier embedded development board, wherein the DeepStream is installed on the Jetson AGX Xavier embedded development board, the DeepStream has a TensorRT-based inference plug-in, the TensorRT-based inference plug-in is used to convert a model file, and a trained lightweight target detection model is deployed on the Jetson AGX Xavier embedded development board, wherein the lightweight target detection model is used for target detection on the multi-channel video streams transmitted by the at least one camera.

[0029] A result display module for receiving the target detection result of the multi-channel video streams transmitted by the embedded device based on the RTSP protocol, and displaying the video with a bounding box.

[0030] In some embodiments, the DeepStream performs target detection on the multi-channel video streams in the form of a pipeline stream, and the processing pipeline stream structure of the DeepStream comprises:

[0031] a video stream acquisition plug-in configured to acquire multiple video streams transmitted by the at least one camera based on an RTSP protocol;

[0032] a codec plug-in including a first dedicated chip for encoding and a second dedicated chip for decoding, the first dedicated chip being configured to perform hardware encoding on the video stream, and the second dedicated chip being configured to perform hardware decoding on the video stream;

[0033] a TensorRT-based inference plug-in configured to convert a weight file of the lightweight target detection model into a TensorRT-supported inference engine, and obtain video structured information and target detection results of the multiple video streams based on the TensorRT-supported inference engine;

[0034] a first output end configured to output the video structured information in a text form;

[0035] a picture superimposition plug-in and a picture splicing plug-in configured to draw a bounding box to a corresponding position of a video based on position information of a target in the target detection results, and splice multiple videos into one picture;

[0036] a second output end configured to output the video processed by the picture superimposition plug-in and the picture splicing plug-in.

[0037] In some embodiments, the lightweight target detection model is an improved VOLOv5s model, and the improved VOLOv5s model is obtained by the following steps:

[0038] replacing a convolution module in a backbone network of the VOLOv5s model with a depth separable convolution module, replacing a Bottleneck module in a neck network of the VOLOv5s model with a GhostBottleneck module, and performing channel pruning on the neck network after the replacement.

[0039] In some embodiments, the channel pruning on the neck network after the replacement includes:

[0040] In a first model training process, a learning rate is periodically changed in a cosine annealing manner based on a preset data set until the model converges to a preset expected position.

[0041] In a second model training process, channels of the neck network after the replacement that completes the first model training process are pruned according to a preset proportion, a data set and hyperparameters are kept unchanged, fine-tuning training is performed, and the channel pruning is completed.

[0042] The application provides a monitoring video analysis method and system based on DeepStream, which deploys a lightweight target detection model to a Jetson AGX Xavier embedded development board through DeepStream and TensorRT, quickly builds an intelligent video monitoring analysis system, and analyzes information in multiple video streams in real time. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0044] Figure 1 Fig. 1 is a hardware structure schematic diagram of a monitoring video analysis system based on DeepStream provided by an embodiment of the present application;

[0045] Figure 2 Fig. 2 is a structure schematic diagram of a DeepStream processing pipeline flow provided by an embodiment of the present application;

[0046] Figure 3 Fig. 3 is one of flow schematic diagrams of a monitoring video analysis method based on DeepStream provided by an embodiment of the present application;

[0047] Figure 4 Fig. 4 is another of flow schematic diagrams of a monitoring video analysis method based on DeepStream provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] With the rapid development of economic society, people's demand for informatization and intelligentization is increasing, and the traditional manual monitoring method has been difficult to meet the monitoring requirements, so the intelligent video monitoring technology emerges as the times require. Intelligent video monitoring is to automatically analyze video monitoring sequences by using computer vision technology without human intervention, automatically identify special targets and behaviors, and automatically give early warning for suspicious events.

[0049] Target detection is the core technology of intelligent video monitoring and an important guarantee for realizing intelligent video monitoring, and has important research significance. The current target detection technology mainly relies on feature extraction of deep neural network to identify the type of object from a picture or a video, and output the position coordinates of the target.

[0050] In one aspect, a video monitoring system usually needs to process data of multiple cameras in real time, and many specific scenarios have requirements for the size of the device. Therefore, in scenarios such as conference rooms and exhibition halls, it is necessary to consider deploying the monitoring video system on an embedded device. Embedded devices generally have good real-time performance and low cost, but the computing power of embedded devices is limited, making it difficult to deploy neural network models. A general solution is to develop a set of bottom operators according to the hardware characteristics of the embedded device, and compile the operator binary file for execution. The results of video stream processing (i.e., bounding boxes) are drawn on the video stream in an overlay manner and displayed in real time on the front end.

[0051] In related technologies, a video monitoring system is built through an embedded device. The front-end embedded device is used to collect video image signals within the monitoring range and compress and encode the video data to facilitate transmission in the network. The network part is composed of modules such as switches and optical fibers, and has the characteristics of low latency. Multiple terminal monitors can simultaneously decode and view the monitoring video. At the same time, the video information is saved through a storage server.

[0052] In order to deploy the model faster and at the same time obtain higher target detection accuracy, the You Only Look Once (YOLO) series of target detection models are generally used in related technologies. Deep learning-based target detection algorithms can be roughly divided into two categories. One is the two-stage method, which divides the whole into two parts, generating candidate frames and identifying objects within the frame, i.e., object recognition and positioning. The other is the one-stage method, which combines object recognition and positioning together and directly gives the detection result. The YOLO series belongs to the one-stage target detection algorithm, which regards the positioning and recognition of the target as a whole, directly regresses the target segmentation bounding box and class probability, and achieves real-time detection.

[0053] The YOLO series makes pictures or videos containing target categories, trains the data set for the model through manual annotation, and trains and detects in a single network. The YOLO series network model is easy to deploy, has strong backbone network feature extraction capability, and can accurately detect the category and position of the target in the inference process.

[0054] In related technologies, algorithms based on YOLOv3 have been improved, using neural network models for object detection in video streams. However, the YOLOv3 neural network is complex, and even if the YOLOv3 backbone is replaced with MobileNetv3 (a lightweight convolutional neural network whose main goal is to reduce model size and computational complexity while maintaining model accuracy), the model's inference still requires significant computing resources. For scenarios with limited space and low latency requirements, information needs to be acquired and processed on embedded devices. Models based on YOLOv3 are slower than advanced detectors when running on embedded devices.

[0055] Therefore, how to deploy target detection models on embedded devices while ensuring the operating speed of the embedded devices is an urgent problem to be solved. Based on this, this invention provides a surveillance video analysis method and system based on DeepStream.

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] Figure 1 This is a schematic diagram of the hardware structure of the DeepStream-based surveillance video analysis system provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the DeepStream-based surveillance video analysis system provided in this embodiment of the invention includes at least:

[0058] At least one camera 101 is used to acquire multiple video streams and transmit the multiple video streams based on the RTSP protocol to an embedded device.

[0059] Specifically, at least one camera 101 simultaneously captures video stream information and transmits it to the embedded device. The camera can be a bullet camera, with a video resolution of 1080P, employing H.264 video encoding and compression technology. The application layer protocol for transmission is the Real-Time Streaming Protocol (RTSP). The RTSP protocol defines how one-to-many applications can efficiently transmit multimedia data over a network; it is an application-level protocol that controls the transmission of real-time data. H.264 video encoding and compression technology can reduce the size of digital video files without affecting image quality, significantly reducing the network bandwidth and storage space required for the video file. In other words, at a specific bitrate, the video image quality is significantly improved.

[0060] Embedded device 102 uses a Jetson AGX Xavier embedded development board. The Jetson AGX Xavier embedded development board is equipped with DeepStream, which comes with a TensorRT-based inference plugin. The TensorRT-based inference plugin is used to convert model files. The Jetson AGX Xavier embedded development board is equipped with a pre-trained lightweight object detection model, which is used to perform object detection on multiple video streams transmitted from at least one camera.

[0061] Specifically, the Jetson AGX Xavier embedded development board is a compact and powerful artificial intelligence (AI) embedded development board. The embedded device in this embodiment of the invention uses the Jetson AGX Xavier embedded development board, which is equipped with DeepStream. DeepStream includes a TensorRT-based inference plugin for converting model files and deploys a trained lightweight object detection model on the embedded device. This trained lightweight object detection model performs object detection on multiple video streams transmitted from at least one camera, quickly providing video surveillance analysis services.

[0062] The result display module 103 is used to receive the target detection results of multiple video streams transmitted by the embedded device 102 based on the RTSP protocol and display the video with bounding boxes.

[0063] Specifically, after the embedded device completes target detection on multiple video streams transmitted from at least one camera, it transmits the target detection results to the network using the RTSP protocol. The target detection results can be displayed in the video as bounding boxes, which are decoded and displayed in real time by the result display module, enabling staff to promptly receive alarm information and take subsequent actions.

[0064] The DeepStream-based video surveillance analysis system provided in this invention deploys a lightweight target detection model on a Jetson AGX Xavier embedded development board using DeepStream and TensorRT, enabling rapid construction of an intelligent video surveillance analysis system that analyzes information from multiple video streams in real time.

[0065] Figure 2 This is a schematic diagram of the DeepStream processing pipeline provided in an embodiment of the present invention, as shown below. Figure 2 As shown, in some embodiments, DeepStream is a stream analysis toolkit based on the open-source GStreamer multimedia framework. The running application is a video stream processing pipeline, where various functions are pieced together as plugins. The DeepStream application has one input and two outputs. DeepStream processes the pipeline stream including:

[0066] This video stream capture plugin is used to capture multiple video streams transmitted from at least one camera based on the RTSP protocol.

[0067] Specifically, the video stream capture plugin acts as the input, using the camera as the main component, and transmits the captured video to the encoding / decoding plugin.

[0068] The encoding / decoding plugin includes a first dedicated chip (NVENC chip) for encoding and a second dedicated chip (NVDEC chip) for decoding. The first dedicated chip is used for hardware encoding of the video stream, and the second dedicated chip is used for hardware decoding of the video stream.

[0069] Specifically, the encoding / decoding plugin contains two dedicated chips for encoding and decoding, respectively. The hardware encoding / decoding solution based on these dedicated chips can make full use of the development board's resources and has higher efficiency compared to software encoding / decoding solutions.

[0070] This TensorRT-based inference plugin converts the weight files of a lightweight object detection model into an inference engine supported by TensorRT, and obtains video structure information and object detection results from multiple video streams based on the TensorRT-supported inference engine.

[0071] Specifically, the TensorRT-based inference plugin is a core component of DeepStream. It first converts the weight file of the lightweight object detection model into an inference engine supported by TensorRT, then reduces computational load through low-precision quantization and network operator fusion. The result of TensorRT inference is structured video information, which is connected to the first output and output as text.

[0072] The first output terminal is used to output the video structured information in text form.

[0073] The image overlay plugin and image stitching plugin are used to draw bounding boxes onto the corresponding positions in the video based on the position information of the target in the target detection results, and stitch multiple video streams into one image.

[0074] Specifically, the image overlay and image stitching plugins draw the target location information from the target detection results onto the corresponding positions in the video as bounding boxes, and then stitch multiple video streams into a single image. The image overlay and image stitching plugins process the video on a frame-by-frame basis. OSD stands for on-screen display, meaning a menu-driven adjustment method.

[0075] The second output is used to output the video processed by the video overlay plugin and the video splicing plugin.

[0076] Specifically, the video processed by the image overlay plugin and the image splicing plugin, after hardware encoding and frame buffering, is output to the network through the second output terminal and displayed in real time in the result display module 103 based on the RTSP protocol.

[0077] In some embodiments, the central processing unit (CPU) and graphics processing unit (GPU) of the Jetson AGX Xavier embedded development board share a high-speed memory, and the video memory buffering process is performed in this high-speed memory.

[0078] In some embodiments, the lightweight object detection model is an improved VOLOv5s model, which is obtained through the following steps:

[0079] Replace the convolutional modules in the VOLOv5s backbone network with depthwise separable convolutional modules, replace the Bottleneck module in the VOLOv5s neck network with the GhostBottleneck module, and perform channel pruning on the replaced neck network.

[0080] Specifically, the lightweight object detection model in this embodiment of the invention can be obtained by improving the VOLOv5s model. The VOLOv5 series of object detection networks includes four types: VOLOv5s, VOLOv5m, VOLOv5l, and VOLOv5x. This embodiment of the invention uses the VOLOv5s model as a base for improvement. The VOLOv5s model is the network with the smallest depth and the smallest feature map width in the VOLOv5 series, resulting in the fastest speed but also the lowest average accuracy. The other three models are based on this model, continuously increasing its depth and width.

[0081] First, depthwise separable convolutions are used to replace the convolution operations in the YOLOv5s backbone network, decoupling spatial and channel features and improving both speed and accuracy. The Ghost Bottleneck module replaces the convolution operations in the Bottleneck module of the VOLOv5s neck network, replacing standard convolution operations with a computationally less expensive linear transformation. To further reduce inference computation overhead, channel pruning is performed on the model's neck section.

[0082] In some embodiments, channel pruning is performed on the replaced neck network, including:

[0083] During the first model training process, based on a pre-set dataset, the learning rate is periodically changed using cosine annealing until the model converges to the pre-set desired position.

[0084] During the second model training process, the channels of the neck network that had completed the first model training process were pruned according to a preset ratio. While keeping the dataset and hyperparameters unchanged, fine-tuning training was performed to complete the channel pruning.

[0085] Specifically, channel pruning requires training the model twice. First, the model is trained using a self-built dataset. During training, cosine annealing is used to reduce the learning rate, and the learning rate is periodically changed to ensure the model converges smoothly to the correct desired position. Then, the neck channels of the trained network are pruned proportionally. Finally, keeping the dataset and hyperparameters constant, fine-tuning is performed. The fine-tuned algorithm is the lightweight object detection model.

[0086] The DeepStream-based surveillance video analysis system provided in this invention has at least the following beneficial effects:

[0087] (1) The system is easy to build, and the coupling between modules is low. The front end uses cameras to capture video, and the back end uses DeepStream's extensible functions to build a web application, realizing an intelligent video monitoring system at low cost. When problems occur in the system, due to the low coupling between modules, it is easy to debug, locate, and troubleshoot the problem. The clearly defined division of labor among the parts also facilitates later maintenance and functional improvement.

[0088] (2) Low latency. On the one hand, because the lightweight target detection model is deployed in an embedded device, the camera that collects video information and the development board that processes the information are physically close, reducing the time overhead required for data transmission. On the other hand, DeepStream's dedicated chip for encoding and decoding uses hardware decoding, which is faster than software encoding and decoding on a server. These two features enable the system to output detection results faster.

[0089] (3) High security. The video stream is processed near the terminal where the information is collected and is not uploaded to the cloud platform, thus avoiding the risk of interception and theft during the upload process.

[0090] In one specific embodiment, four Hikvision bullet cameras are used to capture video at a resolution of 1080P, employing H.264 video encoding and compression technology. The application layer protocol for transmission is the text-based RTSP real-time video streaming protocol. Multiple cameras simultaneously capture video stream information and transmit it to an embedded device. The embedded device uses the Jetson AGX Xavier development platform, leveraging the DeepStream application and TensorRT inference framework to deploy an object detection model (an improved VOLOv5s model) on the development board, enabling rapid development of a video surveillance analysis service. DeepStream decodes and simultaneously performs object detection on multiple video streams in a pipelined streaming manner, then encodes and compresses the processed video before transmitting it over the network via the RTSP protocol. The results display module uses a video player to decode the video and displays the video with bounding boxes on the monitor. Staff can then perform subsequent processing operations on alarm information.

[0091] The improved VOLOv5s model consumes fewer resources and can process video streams from multiple cameras simultaneously on a single embedded device. After deploying the model on the Jetson AGX Xavier development board, it can simultaneously infer four 1080P video streams, each with a frame rate of around 20fps. With the data acquisition device closer to the data processing device, the system boasts better real-time performance and higher processing efficiency.

[0092] Figure 3 This is one of the flowcharts illustrating the DeepStream-based surveillance video analysis method provided in this embodiment of the invention, such as... Figure 3 As shown, the method includes at least the following steps:

[0093] Step 301: Install DeepStream on the Jetson AGX Xavier embedded development board. DeepStream comes with a TensorRT-based inference plugin, which is used to convert model files.

[0094] Step 302: Deploy the trained lightweight object detection model to DeepSteam;

[0095] Step 303: Input the multi-channel video stream transmitted by at least one camera based on the RTSP protocol into the DeepStream video stream acquisition plugin. DeepStream performs target detection on the multi-channel video stream in a pipelined manner.

[0096] Step 304: Encode and compress the processed video stream, and push it to the result display module via the RTSP protocol using a wireless network for display.

[0097] Specifically, DeepStream is installed on the Jetson AGX Xavier embedded development board. DeepStream comes with a TensorRT-based inference plugin, which is used to convert model files. With DeepStream and TensorRT, the trained lightweight object detection model is deployed on the Jetson AGX Xavier embedded development board, quickly building an intelligent video surveillance system. This system receives multiple video streams transmitted from at least one camera in real time via the RTSP protocol. DeepStream performs object detection on the multiple video streams in real time using a pipelined streaming approach. After encoding and compressing the object-detected video streams, they are pushed to the result display module via a wireless network using the RTSP protocol for display.

[0098] The DeepStream-based video surveillance analysis method provided in this invention deploys a lightweight target detection model on a Jetson AGX Xavier embedded development board using DeepStream and TensorRT, quickly building an intelligent video surveillance analysis system to analyze information in multiple video streams in real time.

[0099] In some embodiments, the lightweight object detection model is an improved VOLOv5s model, which is obtained through the following steps:

[0100] Replace the convolutional modules in the VOLOv5s backbone network with depthwise separable convolutional modules, replace the Bottleneck module in the VOLOv5s neck network with the GhostBottleneck module, and perform channel pruning on the replaced neck network.

[0101] Specifically, the lightweight target detection model in this embodiment of the invention can be obtained by improving the VOLOv5s model. First, the convolution operations in the YOLOv5s backbone network are replaced with depthwise separable convolutions to decouple spatial features from channel features, improving both speed and accuracy. The convolution operations in the Bottleneck module of the VOLOv5s neck network are replaced with the Ghost Bottleneck module, replacing standard convolution operations with a computationally less expensive linear transformation. To further reduce inference computation overhead, channel pruning is performed on the model's neck.

[0102] In some embodiments, channel pruning is performed on the replaced neck network, including:

[0103] During the first model training process, based on a pre-set dataset, the learning rate is periodically changed using cosine annealing until the model converges to the pre-set desired position.

[0104] During the second model training process, the channels of the neck network that had completed the first model training process were pruned according to a preset ratio. While keeping the dataset and hyperparameters unchanged, fine-tuning training was performed to complete the channel pruning.

[0105] Specifically, channel pruning requires training the model twice. First, the model is trained using a self-built dataset. During training, cosine annealing is used to reduce the learning rate, and the learning rate is periodically adjusted to ensure the model converges to the correct desired position, completing the first training process. Next, the neck channels of the trained network are pruned proportionally. Finally, keeping the dataset and hyperparameters constant, fine-tuning is performed, completing the second training process. The finely tuned algorithm is the lightweight object detection model.

[0106] In some embodiments, DeepStream performs object detection on multiple video streams in a pipelined manner, including:

[0107] The DeepStream video stream acquisition plugin transmits the received multiple video streams to the DeepStream encoding / decoding plugin, which includes a first dedicated chip for encoding and a second dedicated chip for decoding.

[0108] The hardware-encoded and video-buffered multi-channel video streams are transmitted to the TensorRT-based inference plugin. The TensorRT-based inference plugin then converts the weight file of the lightweight object detection model into an inference engine supported by TensorRT.

[0109] Based on the inference engine supported by TensorRT, the video structure information and object detection results of multiple video streams are obtained;

[0110] The structured information from the video is used as the reasoning result and output as text through the first output terminal of DeepStream;

[0111] Based on the target location information in the target detection results, the bounding box is drawn onto the corresponding position in the video, and multiple video streams are stitched together into one frame. After hardware encoding by the encoding / decoding plugin and buffering in the video memory, the frame is output through the second output terminal of DeepStream.

[0112] Specifically, in combination Figure 2 DeepStream processes multiple video streams using a pipelined flow structure for object detection. Specifically, the video stream acquisition plugin transmits multiple video streams from at least one camera via the RTSP protocol to the DeepStream encoding / decoding plugin. The encoding / decoding plugin includes a first dedicated chip for encoding and a second dedicated chip for decoding. After hardware encoding and memory buffering by the encoding / decoding plugin, the multiple video streams are transmitted to the TensorRT-based inference plugin. The TensorRT-based inference plugin transforms the weight file of the lightweight object detection model into a TensorRT-supported inference engine, obtaining the video structure information and object detection results from the multiple video streams. The video structure information serves as the inference result of the TensorRT-based inference plugin and is output as text through the first output terminal. The object detection results contain the target's location information (e.g., coordinate values). The image overlay and image stitching plugins draw bounding boxes onto the corresponding positions in the video and stitch the multiple video streams into a single image. After hardware encoding and memory buffering by the encoding / decoding plugin, this image is output through the second output terminal.

[0113] In some embodiments, the CPU and GPU of the Jetson AGX Xavier embedded development board share a high-speed memory, and the video memory buffering process is performed in this high-speed memory.

[0114] In some embodiments, object detection results from multiple video streams are obtained based on the inference engine supported by TensorRT, including:

[0115] The inference engine supported by TensorRT performs inference frame by frame. When the lightweight object detection model detects an object in the current video frame, it outputs the object's category and location information as the object detection result.

[0116] If the lightweight object detection model fails to detect the target in the current video frame, it proceeds to the next video frame for detection.

[0117] Specifically, the TensorRT-based inference plugin is the core of DeepStream. After the lightweight object detection model is trained and transformed, the video captured by the camera is provided to TensorRT frame by frame for inference. When the lightweight object detection model detects a target in the current video frame, the model outputs the target's category and location information as the object detection result. Subsequent plugins of DeepStream then draw and stitch together the result to generate a video with bounding boxes, thus generating a video with alarm information. If no target is detected in the current video frame, the detection is performed on the next video frame. This process is repeated continuously, outputting the object detection results in real time so that staff can view the alarm information in real time.

[0118] The technical solution provided by the present invention will be further described below with reference to a specific embodiment. Figure 4 This is the second flowchart of the DeepStream-based surveillance video analysis method provided in this embodiment of the invention, as shown below. Figure 4 As shown, the model is first improved, trained, and transformed. The camera captures video, which is provided frame-by-frame to TensorRT for inference. When the model detects a target in the current frame, it outputs the target's category and location information. Subsequent plugins in DeepStream then render and stitch the video together to create a video with alert information. If the model does not detect a target in the current video frame, it iteratively detects the next frame. This entire detection and rendering process is repeated, outputting the detection results in real time, allowing staff to view real-time alert information.

[0119] It should be noted that the DeepStream-based surveillance video analysis system provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiments and achieve the same technical effect. The two can be referred to each other accordingly.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for analyzing surveillance video based on DeepStream, characterized in that, include: Install DeepStream on the Jetson AGX Xavier embedded development board. DeepStream comes with a TensorRT-based inference plugin, which is used to convert model files. Deploy the trained lightweight object detection model into the DeepStream; At least one camera transmits multiple video streams based on the RTSP protocol, which are then input to the DeepStream video stream acquisition plugin. DeepStream performs target detection on the multiple video streams in a pipelined manner. The processed video stream is encoded and compressed, and then pushed to the result display module for display via a wireless network using the RTSP protocol. The lightweight target detection model is an improved VOLOv5s model, which is obtained through the following steps: Replace the convolutional modules in the VOLOv5s backbone network with depthwise separable convolutional modules, replace the Bottleneck module in the VOLOv5s neck network with GhostBottleneck modules, and perform channel pruning on the replaced neck network. The process of performing channel pruning on the replaced neck network includes: During the first model training process, based on a pre-set dataset, the learning rate is periodically changed using cosine annealing until the model converges to the pre-set desired position. During the second model training process, the channels of the neck network that has been replaced after the first model training process are pruned according to a preset ratio, while keeping the dataset and hyperparameters unchanged, and fine-tuning training is performed to complete the channel pruning. The DeepStream performs target detection on the multiple video streams in a pipelined manner, including: The DeepStream video stream acquisition plugin transmits the received multi-channel video streams to the DeepStream encoding and decoding plugin, which includes a first dedicated chip for encoding and a second dedicated chip for decoding. The multi-channel video streams, after being hardware encoded and buffered in the video memory by the encoding and decoding plugin, are transmitted to the TensorRT-based inference plugin. The TensorRT-based inference plugin then converts the weight file of the lightweight object detection model into an inference engine supported by TensorRT. Based on the inference engine supported by TensorRT, the video structure information and target detection results of the multi-channel video streams are obtained; The structured information of the video is used as the reasoning result and output in text form through the first output terminal of the DeepStream; Based on the target location information in the target detection results, the target is drawn in the form of a bounding box to the corresponding position in the video, and multiple video streams are stitched together into one frame. After hardware encoding by the encoding / decoding plugin and buffering in video memory, the frame is output through the second output terminal of the DeepStream.

2. The surveillance video analysis method based on DeepStream according to claim 1, characterized in that, The target detection results of the multi-video streams are obtained based on the inference engine supported by TensorRT, including: Based on the inference engine supported by TensorRT, inference is performed frame by frame. When the lightweight object detection model detects an object in the current video frame, it outputs the category and location information of the object as the object detection result. If the lightweight target detection model fails to detect a target in the current video frame, it proceeds to the next video frame for detection.

3. The monitoring video analysis method based on DeepStream according to claim 1, characterized in that, The CPU and GPU of the Jetson AGX Xavier embedded development board share a high-speed memory, and the video memory buffering process is performed in the high-speed memory.

4. A surveillance video analysis system based on DeepStream, characterized in that, include: At least one camera is used to capture multiple video streams and transmit the multiple video streams based on the RTSP protocol to an embedded device; The embedded device uses a Jetson AGX Xavier embedded development board, which is equipped with DeepStream. DeepStream comes with a TensorRT-based inference plugin, which is used to convert model files. The Jetson AGX Xavier embedded development board is equipped with a pre-trained lightweight object detection model, which is used to perform object detection on multiple video streams transmitted from at least one camera. The result display module is used to receive the target detection results of the multi-channel video stream transmitted by the embedded device based on the RTSP protocol, and display the video with bounding boxes; The lightweight target detection model is an improved VOLOv5s model, which is obtained through the following steps: Replace the convolutional modules in the VOLOv5s backbone network with depthwise separable convolutional modules, replace the Bottleneck module in the VOLOv5s neck network with GhostBottleneck modules, and perform channel pruning on the replaced neck network. The step of performing channel pruning on the replaced neck network includes: During the first model training process, based on a pre-set dataset, the learning rate is periodically changed using cosine annealing until the model converges to the pre-set desired position. During the second model training process, the channels of the neck network that has been replaced after the first model training process are pruned according to a preset ratio, while keeping the dataset and hyperparameters unchanged, and fine-tuning training is performed to complete the channel pruning. The DeepStream performs target detection on the multiple video streams using a pipelined flow approach. The DeepStream processing pipeline structure includes: A video stream acquisition plugin is used to acquire multiple video streams transmitted by the at least one camera based on the RTSP protocol. The encoding / decoding plugin includes a first dedicated chip for encoding and a second dedicated chip for decoding, wherein the first dedicated chip is used for hardware encoding of the video stream and the second dedicated chip is used for hardware decoding of the video stream; The TensorRT-based inference plugin is used to convert the weight file of the lightweight object detection model into an inference engine supported by TensorRT, and obtain the video structure information and object detection results of the multi-channel video stream based on the inference engine supported by TensorRT. The first output terminal is used to output the video structured information in text form; The image overlay plugin and the image stitching plugin are used to draw bounding boxes onto the corresponding positions in the video based on the position information of the target in the target detection results, and to stitch multiple video streams into one image. The second output terminal is used to output the video processed by the image overlay plugin and the image splicing plugin.

Images