Monitoring video detection and identification method and system based on deep learning

Abstract

The invention relates to the technical field of video detection, in particular to a monitoring video detection and identification method and system based on deep learning, and the method comprises the steps: obtaining a target monitoring video of a monitoring region based on monitoring camera equipment, obtaining a video frame sequence of the target monitoring video, and carrying out the target screening of the video frame sequence, so as to obtain a video frame set; and performing target detection on each video frame in the video frame set based on a trained target detection model to obtain a target detection result set of each video frame in the video frame set. According to the method, through combination of keyword extraction and a large language model, complex monitoring data can be converted into structured event description, the textualized information not only facilitates management personnel to rapidly know the field condition, but also can provide reference for subsequent event processing, and the event description generated in real time is combined, so that the management personnel can conveniently and rapidly know the field condition. And a manager can make a coping strategy more efficiently.

Description

Technical Field

[0001] The present invention relates to the field of video detection technology, and in particular to a surveillance video detection and recognition method and system based on deep learning. Background Art

[0002] Deep learning is a subfield of machine learning that mainly uses artificial neural networks to analyze and process data. It can automatically extract features and perform pattern recognition by simulating the working method of the human brain. Deep learning performs well in many fields, including image recognition, natural language processing, and speech recognition. Surveillance video refers to real-time video captured and recorded by a camera, which is used to monitor activities and events in a specific area.

[0003] Traditional methods usually rely on manual observation, and workers need to stare at the screen for a long time. This method can easily lead to distraction and miss important events or details. Manual monitoring is inefficient and it is difficult to quickly identify and respond to suspicious behaviors or events when processing large amounts of video data, especially when the monitoring area is wide. The response time may be delayed. Traditional methods are usually unable to conduct in-depth data analysis, and it is difficult to extract and integrate valuable information, and it is impossible to achieve a comprehensive understanding and summary of the incident. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a surveillance video detection and recognition method and system based on deep learning.

[0005] The technical solution adopted to solve the above technical problems is: a surveillance video detection and recognition method based on deep learning, including: Acquire a target surveillance video of a surveillance area based on a surveillance camera device, acquire a video frame sequence of the target surveillance video, and perform target screening on the video frame sequence to obtain a video frame set; Performing target detection on each video frame in the video frame set based on the trained target detection model to obtain a target detection result set for each video frame in the video frame set; Based on the target detection result set of each video frame in the video frame set, motion detection is performed on the detection targets in the video frame set to obtain motion trajectory labels of multiple detection targets; Perform keyword extraction on the target detection result set and the motion trajectory labels of the multiple detected targets to obtain a prompt word set, input the prompt word set into a trained large language model, and output an event description set of the target surveillance video based on the trained large language model; A real-time location is acquired based on a GPS module integrated in the monitoring camera device, and a scene detection text set of the target monitoring video is generated based on the real-time location and an event description set of the target monitoring video.

[0006] Preferably, the target detection result set includes target detection frame coordinates, target detection result confidence, target category and target appearance feature information, wherein the target category includes objects, people and environment.

[0007] Preferably, target screening is performed on the video frame sequence to obtain a video frame set, including: The video frame sequence is traversed, and the background of an initial video frame in the video frame sequence is modeled based on Gaussian distribution to obtain a background model corresponding to the initial video frame; When traversing to the current video frame, the background model is compared with the pixel points of the current video frame one by one to classify the pixel points of the current video frame to determine the type of the pixel points, wherein the type of the pixel points includes background points and foreground points; Counting the number of foreground points in the current video frame to obtain the number of foreground points, and calculating a retention coefficient of the current video frame based on the number of foreground points to obtain the retention coefficient of the current video frame; Comparing the retention coefficient of the current video frame with a preset retention threshold, and if the retention coefficient of the current video frame is greater than the preset retention threshold, adding the current video frame to the video frame set; The background model is updated based on the current video frame to obtain a new background model, and the above operation is repeated until all video frames in the video frame sequence are traversed to obtain a video frame set corresponding to the video frame sequence.

[0008] Preferably, the background model is as follows: ; in, Indicates the video frame Moment The pixel value of a pixel, represents the background distribution of pixels, Represents the background model Moment The weights of the Gaussian distribution are Indicates the video frame Moment The pixel The mean of a Gaussian distribution, Indicates the video frame Moment The pixel The covariance matrix of a Gaussian distribution, where , , and Indicates the video frame Moment The average pixel value of the R, G and B components of a pixel in the RGB color space, where , , and Indicates the video frame Moment The standard deviation of the pixel values ​​of the R, G, and B components of a pixel in the RGB color space, represents the probability density function of the Gaussian distribution, where ; The calculation formula of the retention coefficient is as follows: ; in, represents the retention coefficient, represents the background distribution of pixels, Indicates the total number of pixels in the video frame; The updating formula of the background model is as follows: ; in, Indicates whether the pixel is a foreground point. Indicates that the pixel is a foreground point. Indicates that the pixel is a background point. Represents the pixel value of a pixel. and Indicates the preset weight coefficient.

[0009] Preferably, the target detection model extracts features of different scales of the video frame through three efficient convolution modules, which are respectively represented as , and ,Multi-scale feature fusion is used to identify targets based on features of different scales, where for the smallest scale feature , firstly, a 1×1 convolution operation is performed on the minimum scale feature Perform feature transformation and then upsample to obtain Features of the same size, and then compare the feature with The fused features are then input into an efficient convolution module to extract the fused features, and then the fused features are mapped to the same on the same size and place it with Channel-level concatenation is performed, and finally an efficient convolution module is used to extract the final feature, and the feature is input into a classification module to obtain a set of target detection results for the video frame.

[0010] Preferably, the efficient convolution module uses two different branches to process input features, wherein the first branch first extracts the input channel dependency features through a 3×3 depth-level convolution kernel, and then extracts the input spatial dependency features through a 1×1 pixel-level convolution operation, wherein the second branch first extracts the input spatial dependency features through a 1×1 pixel-level convolution operation, and then uses a 3×3 depth-level convolution operation to extract the channel dependency features of the input features, and then uses an attention module to weight each channel to extract the weighted features, and finally uses a 1×1 pixel-level convolution operation to extract the spatial dependency features of the weighted features, and integrates the features of the two branches through channel-level series connection, and integrates the features of the two branches through a 3×3 depth-level convolution operation and a 1×1 pixel-level convolution operation.

[0011] Preferably, performing motion detection on the detected targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain motion trajectory labels of multiple detected targets includes: Traversing the video frame set, comparing the target detection frame coordinates in the target detection result set of the current frame with the target detection frame coordinates of the previous frame in the video frame set, so as to obtain a first dynamic and static state value of the detection target in the current frame; Acquire a second static / dynamic state value of the detected target in the current frame based on the Kalman filter according to the target detection result set, and acquire a final static / dynamic state value of the detected target in the current frame based on the first static / dynamic state value and the second static / dynamic state value; When the final static and dynamic state value of the detection target in the current frame is a dynamic state, obtaining the center coordinates of the target detection frame of the detection target in the current frame, and marking the center coordinates in a preset blank image; The above operation is repeated until the detection target does not exist in the video frame set, so as to mark all center coordinates in the preset blank image.

[0012] Preferably, performing motion detection on the detected targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain motion trajectory labels of multiple detected targets also includes: Calculate the distance between adjacent center coordinates in the preset blank image based on the Euclidean distance formula, and repeat the above operation until all center coordinates in the preset blank image are traversed to obtain the trajectory length of the motion trajectory; Calculating the speed between adjacent center coordinates based on the time interval between adjacent center coordinates in the preset blank image, calculating the acceleration of the motion trajectory based on the speed between adjacent center coordinates, repeating the above operation until all center coordinates in the preset blank image are traversed to obtain the acceleration sequence of the motion trajectory; Calculate the direction angles between adjacent center coordinates in the preset blank image, and repeat the above operation until all center coordinates in the preset blank image are traversed to obtain a direction angle sequence of the motion trajectory; Encoding the trajectory length, acceleration sequence and direction angle sequence of the motion trajectory to obtain a feature vector of the motion trajectory; The feature vector of the motion trajectory is classified based on the trained trajectory classification model to obtain the motion trajectory label of the detection target.

[0013] Preferably, the calculation formula of the final dynamic and static state value is as follows: ; in, Represents the final dynamic and static state value, Indicates that the final dynamic and static state value is the static state, Indicates that the final static and dynamic state value is a dynamic state. Indicates the first dynamic and static state value, Indicates the second dynamic and static state value.

[0014] Preferably, the method further comprises: Constructing a monitoring graph structure corresponding to the monitoring area based on the target detection result set of each video frame in the video frame set, wherein the monitoring graph structure includes a node set and an edge set, wherein the nodes in the node set correspond to the monitoring area, and the edges in the edge set correspond to the spatial adjacent relationship, wherein the spatial adjacent relationship is used to indicate that the detection target is transferred from one monitoring area to another monitoring area; Anomaly detection is performed on the monitoring graph structure based on a pre-trained region detection model to obtain anomaly detection labels corresponding to the monitoring area, wherein the region detection model adopts a graph neural network, and the update formula of the graph neural network is as follows: ; in, Representation Node In the The feature vector of the layer, Represents the graph neural network The weight matrix of the layer, represents the activation function, Representation Node and neighbor nodes The edge weights between Represents a set of neighbor nodes.

[0015] The technical solution adopted to solve the above technical problems is: a surveillance video detection and recognition system based on deep learning, which is applicable to the surveillance video detection and recognition method based on deep learning, including: A video frame screening unit, the video frame screening unit is used to obtain a target monitoring video of a monitoring area based on a monitoring camera device, obtain a video frame sequence of the target monitoring video, and perform target screening on the video frame sequence to obtain a video frame set; An object detection unit, the object detection unit is used to perform object detection on each video frame in the video frame set based on a trained object detection model to obtain an object detection result set for each video frame in the video frame set; A motion detection unit, the motion detection unit being used to perform motion detection on a detection target in the video frame set based on a target detection result set of each video frame in the video frame set, so as to obtain motion trajectory labels of a plurality of detection targets; An event description unit, the event description unit is used to perform keyword extraction on the target detection result set and the motion trajectory labels of the multiple detection targets to obtain a prompt word set, input the prompt word set into a trained large language model, and output an event description set of the target surveillance video based on the trained large language model; A text generation unit is used to obtain a real-time position based on a GPS module integrated in the monitoring camera device, and to generate a scene detection text set of the target monitoring video based on the real-time position and an event description set of the target monitoring video.

[0016] The beneficial effects of the present invention are as follows: (1) The present invention significantly improves the working efficiency of video surveillance through an automated video analysis process. After adopting the target detection model and motion analysis technology, the video frames can be processed in real time, and the targets can be automatically identified and tracked. This process reduces the investment in human resources, making monitoring more efficient. Moreover, through the trained target detection model, various targets can be accurately identified in the video frames, and detailed behavior analysis can be performed. (2) The present invention can track the trajectory of the target through motion detection technology, and provide data support for identifying suspicious behaviors. This accuracy is particularly important in the fields of public safety and traffic management, and can effectively prevent and respond to potential security threats. Moreover, through the combination of keyword extraction and large language models, complex monitoring data can be converted into structured event descriptions. This textual information not only facilitates management personnel to quickly understand the on-site conditions, but also provides for subsequent event processing. Reference, combined with the event description generated in real time, managers can formulate response strategies more efficiently; (3) The present invention integrates the function of the GPS module, which can obtain the real-time location and combine it with the monitoring event to generate scene detection text. This integration of spatial information provides a more comprehensive background for event analysis, helps to enhance the understanding of the on-site situation, and improves the accuracy of event processing. By constructing a monitoring graph structure and applying graph neural network (GNN) for anomaly detection, it can more efficiently identify and track the transfer of targets between different monitoring areas. This method can capture the spatial relationship of the target and detect abnormal behavior based on the structural update of the monitoring graph, thereby improving the accuracy and response speed of the monitoring system. Combining the target detection results with the advantages of graph neural networks, it realizes automated monitoring area anomaly detection without human intervention, and can be deployed on a large scale in different scenarios, improving the intelligence and efficiency of the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic flow chart of the steps of an overall method in an embodiment of the present invention; Figure 2 A schematic diagram of the system architecture of an overall system in an embodiment of the present invention.

[0018] Figure numerals: 1. Video frame screening unit; 2. Target detection unit; 3. Motion detection unit; 4. Event description unit; 5. Text generation unit. DETAILED DESCRIPTION

[0019] Embodiment 1, as Figure 1 As shown, the surveillance video detection and recognition method based on deep learning proposed by the present invention includes: S1. Obtaining a target surveillance video of a surveillance area based on a surveillance camera device, obtaining a video frame sequence of the target surveillance video, and performing target screening on the video frame sequence to obtain a video frame set; S2. performing target detection on each video frame in the video frame set based on the trained target detection model to obtain a target detection result set for each video frame in the video frame set; S3, performing motion detection on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain motion trajectory labels of multiple detection targets; S4, performing keyword extraction on the target detection result set and the motion trajectory labels of the multiple detected targets to obtain a prompt word set, inputting the prompt word set into a trained large language model, and outputting an event description set of the target surveillance video based on the trained large language model; S5. Acquire the real-time location based on the GPS module integrated in the monitoring camera device, and generate a scene detection text set of the target monitoring video based on the real-time location and the event description set of the target monitoring video.

[0020] In the present invention, the monitoring camera equipment is used to capture the monitoring area in real time; the video frame sequence means that the monitoring video is composed of a series of continuous static images (frames), and the video frame sequence refers to the collection of these static images; target screening refers to extracting specific objects (such as pedestrians, vehicles, etc.) from video frames to generate a video frame collection; keyword extraction refers to extracting important words or phrases from target detection results and motion trajectory labels, usually using natural language processing technology; the large language model (LLM) is used to generate a deep learning model for text, and generates relevant event descriptions based on the prompt word set; the GPS module is a global positioning system module integrated in the monitoring device, which is used to obtain the real-time location information of the device; the scene detection text collection refers to the text generated based on the real-time location and event description, usually including a specific description of the scene, the name of the place, etc.

[0021] Embodiment 2: The surveillance video detection and recognition method based on deep learning proposed in the present invention, compared with embodiment 1, this embodiment also includes: the target detection result set includes target detection frame coordinates, target detection result confidence, target category and target appearance feature information, wherein the target category includes objects, people and environment.

[0022] In an optional embodiment, target screening is performed on a video frame sequence to obtain a video frame set, including: A1. Traversing the video frame sequence, modeling the background of the initial video frame in the video frame sequence based on Gaussian distribution to obtain a background model corresponding to the initial video frame; A2. When traversing to the current video frame, the background model is compared with the pixels of the current video frame one by one to classify the pixels of the current video frame to determine the type of the pixels, wherein the types of the pixels include background points and foreground points; A3, counting the number of foreground points in the current video frame to obtain the number of foreground points, and calculating the retention coefficient of the current video frame based on the number of foreground points to obtain the retention coefficient of the current video frame; A4, comparing the retention coefficient of the current video frame with a preset retention threshold, if the retention coefficient of the current video frame is greater than the preset retention threshold, adding the current video frame to the video frame set; A5. Update the background model based on the current video frame to obtain a new background model, and repeat the above operation until all video frames in the video frame sequence are traversed to obtain a video frame set corresponding to the video frame sequence.

[0023] It should be noted that Gaussian distribution (or normal distribution) is a common probability distribution used to describe many natural phenomena. In video processing, Gaussian distribution can be used to model the color value distribution of background pixels to distinguish between background and foreground. The background model is a mathematical representation of the unchanging background in the video, which is used to determine which pixels are the background (static part) and which are the foreground (dynamic part). It is constructed by statistically analyzing the pixel values ​​in multiple frames. The retention coefficient is an indicator used to quantify the importance of the current video frame. It is usually calculated based on the number of foreground points in the current frame. The higher the retention coefficient, the more important the information contained in the current frame. After each frame is processed, background model update refers to adjusting the background model according to the classification results of the foreground and background of the current frame to make it more accurately reflect changes in the current environment. This step is to improve the processing accuracy of subsequent frames.

[0024] In an optional embodiment, the background model is as follows: ; in, Indicates the video frame Moment The pixel value of a pixel, represents the background distribution of pixels, Represents the background model Moment The weights of the Gaussian distribution are Indicates the video frame Moment The pixel The mean of a Gaussian distribution, Indicates the video frame Moment The pixel The covariance matrix of a Gaussian distribution, where , , and Indicates the video frame Moment The average pixel value of the R, G and B components of a pixel in the RGB color space, where , , and Indicates the video frame Moment The standard deviation of the pixel values ​​of the R, G, and B components of a pixel in the RGB color space, represents the probability density function of the Gaussian distribution, where ; The retention coefficient is calculated as follows: ; in, represents the retention coefficient, represents the background distribution of pixels, Indicates the total number of pixels in the video frame; The update formula of the background model is as follows: ; in, Indicates whether the pixel is a foreground point. Indicates that the pixel is a foreground point. Indicates that the pixel is a background point. Represents the pixel value of a pixel. and Indicates the preset weight coefficient.

[0025] In an optional embodiment, the target detection model extracts features of different scales of video frames through three efficient convolution modules, which are represented as , and ,Multi-scale feature fusion is used to identify targets based on features of different scales, where for the smallest scale feature , firstly, a 1×1 convolution operation is performed on the minimum scale feature Perform feature transformation and then upsample to obtain Features of the same size, and then compare the feature with The fused features are then input into an efficient convolution module to extract the fused features, and then the fused features are mapped to the same on the same size and place it with Channel-level concatenation is performed, and finally an efficient convolution module is used to extract the final feature, which is then input into a classification module to obtain a set of target detection results for the video frame.

[0026] It should be noted that the efficient convolution module refers to an optimized structure used in convolutional neural networks (CNNs) to improve the efficiency and effectiveness of convolution operations. These modules usually contain multiple convolution operations, such as depthwise separable convolutions, which can reduce the amount of computation while maintaining the ability to extract features; multi-scale feature fusion combines features from different scales (sizes) to improve the accuracy of target detection, which means that the model can better identify targets of various sizes because features of different scales provide different levels of information; upsampling is an operation that increases the size of a feature map, which is usually used to restore the spatial information of a feature map.

[0027] In an optional embodiment, the efficient convolution module uses two different branches to process input features, wherein the first branch first extracts the input channel dependency features through a 3×3 depth-level convolution kernel, and then extracts the input spatial dependency features through a 1×1 pixel-level convolution operation, wherein the second branch first extracts the input spatial dependency features through a 1×1 pixel-level convolution operation, and then uses a 3×3 depth-level convolution operation to extract the channel dependency features of the input features, and then uses an attention module to weight each channel to extract the weighted features, and finally uses a 1×1 pixel-level convolution operation to extract the spatial dependency features of the weighted features, and integrates the features of the two branches through channel-level series connection, and integrates the features of the two branches through a 3×3 depth-level convolution operation and a 1×1 pixel-level convolution operation.

[0028] It should be noted that spatial dependency features refer to the relationship between the features of a certain position and the features of the surrounding positions on the feature map. This dependency can help the model understand the structure and patterns in the local area; channel-level concatenation is to splice feature maps from different branches in the channel dimension, which can effectively integrate features from different sources and enhance the network's utilization of information.

[0029] In an optional embodiment, motion detection is performed on the detected targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain motion trajectory labels of multiple detected targets, including: B1. Traverse the video frame set, compare the target detection frame coordinates in the target detection result set of the current frame with the target detection frame coordinates of the previous frame in the video frame set, so as to obtain a first dynamic and static state value of the detection target in the current frame; B2. Obtaining a second dynamic and static state value of the detection target in the current frame based on the Kalman filter according to the target detection result set, and obtaining a final dynamic and static state value of the detection target in the current frame based on the first dynamic and static state value and the second dynamic and static state value; B4. When the final static and dynamic state value of the detection target in the current frame is a dynamic state, the center coordinates of the target detection frame of the detection target in the current frame are obtained, and the center coordinates are marked in a preset blank image; B5. Repeat the above operation until there is no detection target in the video frame set, so as to mark all center coordinates in the preset blank image.

[0030] It should be noted that Kalman filtering is a recursive algorithm that is often used to estimate the state of a dynamic system. In target tracking, Kalman filtering can be used to predict the next position of the target while integrating current observation information to reduce the impact of noise.

[0031] In an optional embodiment, performing motion detection on the detected targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain motion trajectory labels of multiple detected targets also includes: B6. Calculate the distance between adjacent center coordinates in the preset blank image based on the Euclidean distance formula, and repeat the above operation until all center coordinates in the preset blank image are traversed to obtain the trajectory length of the motion trajectory; B7, calculating the speed between adjacent center coordinates based on the time interval between adjacent center coordinates in the preset blank image, and calculating the acceleration of the motion trajectory based on the speed between adjacent center coordinates, repeating the above operation until all center coordinates in the preset blank image are traversed to obtain the acceleration sequence of the motion trajectory; B8. Calculate the direction angles between adjacent center coordinates in the preset blank image, and repeat the above operation until all center coordinates in the preset blank image are traversed to obtain the direction angle sequence of the motion trajectory; B9, encoding the trajectory length, acceleration sequence and direction angle sequence of the motion trajectory to obtain a feature vector of the motion trajectory; B10. Classify the feature vector of the motion trajectory based on the trained trajectory classification model to obtain the motion trajectory label of the detection target.

[0032] It should be noted that the trajectory classification model uses a support vector machine; the motion trajectory labels include abnormal labels and normal labels.

[0033] In an optional embodiment, the calculation formula of the final dynamic and static state value is as follows: ; in, Represents the final dynamic and static state value, Indicates that the final dynamic and static state value is the static state, Indicates that the final static and dynamic state value is a dynamic state. Indicates the first dynamic and static state value, Indicates the second dynamic and static state value.

[0034] It should be noted that each frame in the video frame set is traversed, and the target detection frame of the current frame is compared with the target detection frame of the previous frame. By comparing the position of the target detection frame (usually the center coordinates of the detection frame), the state of the target in the current frame compared with the previous frame (dynamic or static) can be obtained. If the position of the target detection frame changes significantly, it is judged as a dynamic target; if the detection frame hardly changes, it is judged as a static target. Kalman filter is a dynamic system estimation method based on statistics, which is usually used to estimate the state of an object in a noisy environment. In target detection, Kalman filter is usually used to predict the target position. The Kalman filter predicts the position of the target in the current frame based on the state of the previous frame. Based on the result of the Kalman filter, the Kalman filter predicts the target position in the current frame according to the target's motion trend. If there is a significant difference between the predicted position of the Kalman filter and the actually detected position, this may indicate that the target is moving. If the difference between the two is very small, it can be considered that the target is in a stationary state. The final dynamic and static state value is determined by combining the first dynamic and static state value and the second dynamic and static state value. Usually, if the first dynamic and static state value and the second dynamic and static state value are consistent (for example, both are expressed as "dynamic" or "static"), the final dynamic and static state value takes a consistent state.

[0035] In an optional embodiment, the method further includes: Based on the target detection result set of each video frame in the video frame set, a monitoring graph structure corresponding to the monitoring area is constructed, wherein the monitoring graph structure includes a node set and an edge set, wherein the nodes in the node set correspond to the monitoring areas, and the edges in the edge set correspond to the spatial adjacent relationships, wherein the spatial adjacent relationships are used to indicate that the detection target is transferred from one monitoring area to another monitoring area; Based on the pre-trained region detection model, anomaly detection is performed on the monitoring graph structure to obtain the anomaly detection label corresponding to the monitoring area. The region detection model uses a graph neural network, and the update formula of the graph neural network is as follows: ; in, Representation Node In the The feature vector of the layer, Represents the graph neural network The weight matrix of the layer, represents the activation function, Representation Node and neighbor nodes The edge weights between Represents a set of neighbor nodes.

[0036] It should be noted that Graph Neural Network is a deep learning model that specializes in processing graph structured data. Unlike traditional neural networks, GNN can capture the relationship information between nodes in graph structured data and update the features of each node through a message passing mechanism.

[0037] Embodiment three, as Figure 2 As shown, the surveillance video detection and recognition system based on deep learning proposed by the present invention is applicable to the surveillance video detection and recognition method based on deep learning, including: The video frame screening unit 1 is used to obtain a target surveillance video of a surveillance area based on a surveillance camera device, obtain a video frame sequence of the target surveillance video, and perform target screening on the video frame sequence to obtain a video frame set; The target detection unit 2 is used to perform target detection on each video frame in the video frame set based on the trained target detection model to obtain a target detection result set for each video frame in the video frame set; A motion detection unit 3, the motion detection unit 3 is used to perform motion detection on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set, so as to obtain motion trajectory labels of multiple detection targets; The event description unit 4 is used to extract keywords from the target detection result set and the motion trajectory labels of multiple detection targets to obtain a prompt word set, input the prompt word set into the trained large language model, and output the event description set of the target monitoring video based on the trained large language model; The text generation unit 5 is used to obtain the real-time position based on the GPS module integrated in the monitoring camera device, and generate a scene detection text set of the target monitoring video based on the real-time position and the event description set of the target monitoring video.

[0038] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A surveillance video detection and recognition method based on deep learning, characterized in that: include: Acquire a target surveillance video of a surveillance area based on a surveillance camera device, acquire a video frame sequence of the target surveillance video, and perform target screening on the video frame sequence to obtain a video frame set; Performing target detection on each video frame in the video frame set based on the trained target detection model to obtain a target detection result set for each video frame in the video frame set; Based on the target detection result set of each video frame in the video frame set, motion detection is performed on the detection targets in the video frame set to obtain motion trajectory labels of multiple detection targets; Perform keyword extraction on the target detection result set and the motion trajectory labels of the multiple detected targets to obtain a prompt word set, input the prompt word set into a trained large language model, and output an event description set of the target surveillance video based on the trained large language model; A real-time location is acquired based on a GPS module integrated in the monitoring camera device, and a scene detection text set of the target monitoring video is generated based on the real-time location and an event description set of the target monitoring video.

2. The surveillance video detection and recognition method based on deep learning according to claim 1 is characterized in that: The target detection result set includes target detection frame coordinates, target detection result confidence, target category and target appearance feature information, wherein the target category includes objects, people and environment.

3. The surveillance video detection and recognition method based on deep learning according to claim 2 is characterized in that: Performing target screening on the video frame sequence to obtain a video frame set includes: The video frame sequence is traversed, and the background of an initial video frame in the video frame sequence is modeled based on Gaussian distribution to obtain a background model corresponding to the initial video frame; When traversing to the current video frame, the background model is compared with the pixel points of the current video frame one by one to classify the pixel points of the current video frame to determine the type of the pixel points, wherein the type of the pixel points includes background points and foreground points; Counting the number of foreground points in the current video frame to obtain the number of foreground points, and calculating a retention coefficient of the current video frame based on the number of foreground points to obtain the retention coefficient of the current video frame; Comparing the retention coefficient of the current video frame with a preset retention threshold, and if the retention coefficient of the current video frame is greater than the preset retention threshold, adding the current video frame to the video frame set; The background model is updated based on the current video frame to obtain a new background model, and the above operation is repeated until all video frames in the video frame sequence are traversed to obtain a video frame set corresponding to the video frame sequence.

4. The surveillance video detection and recognition method based on deep learning according to claim 3 is characterized in that: The background model is as follows: ； in, Indicates the video frame Moment The pixel value of a pixel, represents the background distribution of pixels, Represents the background model Moment The weights of the Gaussian distribution are Indicates the video frame Moment The pixel The mean of a Gaussian distribution, Indicates the video frame Moment The pixel The covariance matrix of a Gaussian distribution, where , , and Indicates the video frame Moment The average pixel value of the R, G and B components of a pixel in the RGB color space, where , , and Indicates the video frame Moment The standard deviation of the pixel values ​​of the R, G, and B components of a pixel in the RGB color space, represents the probability density function of the Gaussian distribution, where ; The calculation formula of the retention coefficient is as follows: ； in, represents the retention coefficient, represents the background distribution of pixels, Indicates the total number of pixels in the video frame; The updating formula of the background model is as follows: ； in, Indicates whether the pixel is a foreground point. Indicates that the pixel is a foreground point. Indicates that the pixel is a background point. Represents the pixel value of a pixel. and Indicates the preset weight coefficient.

5. The surveillance video detection and recognition method based on deep learning according to claim 2 is characterized in that: The target detection model extracts features of different scales of the video frame through three efficient convolution modules, which are represented as , and ,Multi-scale feature fusion is used to identify targets based on features of different scales, where for the smallest scale feature First, a 1×1 convolution operation is performed on the minimum scale feature Perform feature transformation and then upsample to obtain Features of the same size, and then compare the feature with The fused features are then input into an efficient convolution module to extract the fused features, and then the fused features are mapped to the same on the same size and place it with Channel-level concatenation is performed, and finally an efficient convolution module is used to extract the final feature, and the feature is input into a classification module to obtain a set of target detection results for the video frame.

6. The surveillance video detection and recognition method based on deep learning according to claim 5 is characterized in that: The efficient convolution module uses two different branches to process input features, wherein the first branch first extracts the input channel dependency features through a 3×3 depth-level convolution kernel, and then extracts the input spatial dependency features through a 1×1 pixel-level convolution operation, wherein the second branch first extracts the input spatial dependency features through a 1×1 pixel-level convolution operation, and then uses a 3×3 depth-level convolution operation to extract the channel dependency features of the input features, and then uses an attention module to weight each channel to extract the weighted features, and finally uses a 1×1 pixel-level convolution operation to extract the spatial dependency features of the weighted features, and integrates the features of the two branches through channel-level cascade, and integrates the features of the two branches through a 3×3 depth-level convolution operation and a 1×1 pixel-level convolution operation.

7. The surveillance video detection and recognition method based on deep learning according to claim 1 is characterized in that: Performing motion detection on the detected targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain motion trajectory labels of multiple detected targets, including: Traversing the video frame set, comparing the target detection frame coordinates in the target detection result set of the current frame with the target detection frame coordinates of the previous frame in the video frame set, so as to obtain a first dynamic and static state value of the detection target in the current frame; According to the target detection result set, a second dynamic and static state value of the detection target in the current frame is obtained based on Kalman filtering, and a final dynamic and static state value of the detection target in the current frame is obtained based on the first dynamic and static state value and the second dynamic and static state value, wherein the calculation formula of the final dynamic and static state value is as follows: ； in, Represents the final dynamic and static state value, Indicates that the final dynamic and static state value is the static state, Indicates that the final static and dynamic state value is a dynamic state. Indicates the first dynamic and static state value, Indicates the second dynamic and static state value; When the final static and dynamic state value of the detection target in the current frame is a dynamic state, obtaining the center coordinates of the target detection frame of the detection target in the current frame, and marking the center coordinates in a preset blank image; The above operation is repeated until the detection target does not exist in the video frame set, so as to mark all center coordinates in the preset blank image.

8. The surveillance video detection and recognition method based on deep learning according to claim 7 is characterized in that: Performing motion detection on the detected targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain motion trajectory labels of multiple detected targets, further comprising: Calculate the distance between adjacent center coordinates in the preset blank image based on the Euclidean distance formula, and repeat the above operation until all center coordinates in the preset blank image are traversed to obtain the trajectory length of the motion trajectory; Calculating the speed between adjacent center coordinates based on the time interval between adjacent center coordinates in the preset blank image, calculating the acceleration of the motion trajectory based on the speed between adjacent center coordinates, repeating the above operation until all center coordinates in the preset blank image are traversed to obtain the acceleration sequence of the motion trajectory; Calculate the direction angles between adjacent center coordinates in the preset blank image, and repeat the above operation until all center coordinates in the preset blank image are traversed to obtain a direction angle sequence of the motion trajectory; Encoding the trajectory length, acceleration sequence and direction angle sequence of the motion trajectory to obtain a feature vector of the motion trajectory; The feature vector of the motion trajectory is classified based on the trained trajectory classification model to obtain the motion trajectory label of the detection target.

9. The surveillance video detection and recognition method based on deep learning according to claim 2 is characterized in that: The method further comprises: Constructing a monitoring graph structure corresponding to the monitoring area based on the target detection result set of each video frame in the video frame set, wherein the monitoring graph structure includes a node set and an edge set, wherein the nodes in the node set correspond to the monitoring area, and the edges in the edge set correspond to the spatial adjacent relationship, wherein the spatial adjacent relationship is used to indicate that the detection target is transferred from one monitoring area to another monitoring area; Anomaly detection is performed on the monitoring graph structure based on a pre-trained region detection model to obtain anomaly detection labels corresponding to the monitoring area, wherein the region detection model adopts a graph neural network, and the update formula of the graph neural network is as follows: ； in, Representation Node In the The feature vector of the layer, Represents the graph neural network The weight matrix of the layer, represents the activation function, Representation Node and neighbor nodes The edge weights between Represents a set of neighbor nodes.

10. A surveillance video detection and recognition system based on deep learning, which is applicable to the surveillance video detection and recognition method based on deep learning according to any one of claims 1 to 9, characterized in that: include: A video frame screening unit (1), the video frame screening unit (1) is used to obtain a target monitoring video of a monitoring area based on a monitoring camera device, obtain a video frame sequence of the target monitoring video, and perform target screening on the video frame sequence to obtain a video frame set; An object detection unit (2), the object detection unit (2) being used to perform object detection on each video frame in the video frame set based on a trained object detection model, so as to obtain an object detection result set for each video frame in the video frame set; A motion detection unit (3), the motion detection unit (3) being used to perform motion detection on a detection target in the video frame set based on a target detection result set of each video frame in the video frame set, so as to obtain motion trajectory labels of a plurality of detection targets; An event description unit (4), the event description unit (4) being used to perform keyword extraction on the target detection result set and the motion trajectory labels of the plurality of detected targets to obtain a prompt word set, input the prompt word set into a trained large language model, and output an event description set of the target surveillance video based on the trained large language model; A text generation unit (5), the text generation unit (5) is used to obtain a real-time position based on a GPS module integrated in the monitoring camera device, and to generate a scene detection text set of the target monitoring video based on the real-time position and an event description set of the target monitoring video.

Images