Deep Learning-Based Monitoring Video Detection and Recognition Method and System

Abstract

The present invention relates to the field of video detection technology, and specifically to a method and system for detecting and recognizing surveillance videos based on deep learning, including: obtaining a target surveillance video of a surveillance area based on a surveillance camera device, obtaining a video frame sequence of the target surveillance video, and performing target screening on the video frame sequence to obtain a video frame set; performing target detection on each video frame in the video frame set based on a trained target detection model to obtain a target detection result set for each video frame in the video frame set. Through the combination of keyword extraction and a large language model, the present invention can convert complex surveillance data into structured event descriptions. Such textual information not only facilitates managers to quickly understand the on-site situation, but also provides a reference for subsequent event processing. Combining with the real-time generated event descriptions, managers can formulate response strategies more efficiently.

Description

Technical Field

[0001] The present invention relates to the technical field of video detection, and particularly to a method and system for monitoring video detection and recognition based on deep learning. Background Art

[0002] Deep learning is a subfield of machine learning that mainly uses artificial neural networks to analyze and process data. By simulating the working mode of the human brain, it can automatically extract features and perform pattern recognition. Deep learning has performed excellently in many fields, including image recognition, natural language processing, and speech recognition, etc. Monitoring video refers to the real-time video captured and recorded by a camera for monitoring activities and events in a specific area.

[0003] Traditional methods usually rely on manual observation. Staff need to stare at the screen for a long time, which is prone to distraction and easy to miss important events or details. The efficiency of manual monitoring is low. When dealing with a large amount of video data, it is difficult to quickly identify and respond to suspicious behaviors or events, especially in the case of a wide monitoring area, the response time may be delayed; and traditional methods usually cannot perform in-depth data analysis, it is difficult to extract and integrate valuable information, and it is impossible to achieve a comprehensive understanding and summary of events. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a method and system for monitoring video detection and recognition based on deep learning.

[0005] The technical solution adopted to solve the above technical problem is: A method for monitoring video detection and recognition based on deep learning, including:

[0006] Obtaining a target monitoring video of a monitoring area based on a monitoring camera device, obtaining a video frame sequence of the target monitoring video, and performing target screening on the video frame sequence to obtain a video frame set;

[0007] Performing target detection on each video frame in the video frame set based on a trained target detection model to obtain a target detection result set for each video frame in the video frame set;

[0008] Performing motion detection on the detection targets in the video frame set based on the target detection result set for each video frame in the video frame set to obtain motion trajectory labels of multiple detection targets;

[0009] Performing keyword extraction on the target detection result set and the motion trajectory labels of the multiple detection targets to obtain a set of prompt words, inputting the set of prompt words into a trained large language model, and outputting a set of event descriptions of the target monitoring video based on the trained large language model;

[0010] Obtain the real-time location based on the GPS module integrated in the monitoring camera device, and generate a scene detection text set of the target monitoring video based on the real-time location and the event description set of the target monitoring video.

[0011] Preferably, the target detection result set includes target detection box coordinates, target detection result confidence, target category, and target appearance feature information, where the target category includes objects, people, and the environment.

[0012] Preferably, perform target screening on the video frame sequence to obtain a video frame set, including:

[0013] Traverse the video frame sequence, model the background of the initial video frame in the video frame sequence based on the Gaussian distribution to obtain the background model corresponding to the initial video frame;

[0014] When traversing to the current video frame, compare each pixel point of the current video frame with the background model one by one to classify the pixel points of the current video frame to determine the type of the pixel points, where the type of the pixel points includes background points and foreground points;

[0015] Count the number of foreground points determined in the current video frame to obtain the foreground point number, and calculate the retention coefficient of the current video frame based on the foreground point number to obtain the retention coefficient of the current video frame;

[0016] Compare the retention coefficient of the current video frame with a preset retention threshold. If the retention coefficient of the current video frame is greater than the preset retention threshold, add the current video frame to the video frame set;

[0017] Update the background model based on the current video frame to obtain a new background model, and repeat the above operations until all video frames in the video frame sequence are traversed to obtain the video frame set corresponding to the video frame sequence.

[0018] Preferably, the background model is as follows:

[0019]

[0020] Where x j,t represents the pixel value of the j-th pixel point at time t in the video frame, P represents the background distribution of the pixel point, represents the weight value of the i-th Gaussian distribution at time t in the background model, represents the average value of the i-th Gaussian distribution of the j-th pixel point at time t in the video frame, represents the covariance matrix of the i-th Gaussian distribution of the j-th pixel at time t in the video frame, where, and represents the average pixel values of the R, G, and B components of the j-th pixel at time t in the video frame in the RGB color space, where, and represents the standard deviation of the pixel values of the R, G, and B components of the j-th pixel at time t in the video frame in the RGB color space, η represents the probability density function of the Gaussian distribution, where,

[0021] The calculation formula of the retention coefficient is as follows:

[0022]

[0023] where, τ represents the retention coefficient, q represents the number of foreground pixels in the video frame, and m*n represents the total number of pixels in the video frame;

[0024] The update formula of the background model is as follows:

[0025]

[0026] where, M i,t represents whether the pixel is a foreground point, M i,t =1 indicates that the pixel is a foreground point, M i,t =0 indicates that the pixel is a background point, X t represents the pixel value of the pixel, and α and ρ represent preset weight coefficients.

[0027] Preferably, the target detection model extracts features of different scales of the video frame through 3 efficient convolution modules, respectively denoted as C1, C2, and C3, and uses multi-scale feature fusion to perform target recognition on features of different scales. Among them, for the smallest-scale feature C3, first perform a feature transformation on the smallest-scale feature C3 through a 1×1 convolution operation, then perform an upsampling operation to obtain a feature of the same size as C2, then fuse this feature with C2, and input it into an efficient convolution module to extract the fused feature. Then, map the fused feature to the same size as C1 through an upsampling operation, and perform channel-level concatenation with C1. Finally, use an efficient convolution module to extract the final feature, and input this feature into a classification module to obtain the target detection result set of the video frame.

[0028] Preferably, the efficient convolution module processes the input features using two different branches. Among them, the first branch first extracts the input channel - dependent features through a 3×3 depth - wise convolution kernel, and then extracts the input spatial - dependent features through a 1×1 pixel - wise convolution operation. Among them, the second branch first extracts the input spatial - dependent features through a 1×1 pixel - wise convolution operation, then uses a 3×3 depth convolution operation to extract the channel - dependent features of the input features, and then weights each channel through an attention module to extract the weighted features. Finally, it extracts the spatial - dependent features of the weighted features through a 1×1 pixel - wise convolution operation. The features of the two branches are integrated in a channel - level concatenation manner, and the features of the two branches are further integrated through a 3×3 depth convolution operation and a 1×1 pixel - wise convolution operation.

[0029] Preferably, motion detection is performed on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain the motion trajectory labels of multiple detection targets, including:

[0030] Traverse the video frame set, compare the target detection box coordinates in the target detection result set of the current frame with the target detection box coordinates of the previous frame in the video frame set to obtain the first static - dynamic state value of the detection target in the current frame;

[0031] Based on the target detection result set, obtain the second static - dynamic state value of the detection target in the current frame through Kalman filtering, and obtain the final static - dynamic state value of the detection target in the current frame based on the first static - dynamic state value and the second static - dynamic state value;

[0032] When the final static - dynamic state value of the detection target in the current frame is a dynamic state, obtain the center coordinates of the target detection box of the detection target in the current frame, and mark the center coordinates in a preset blank image;

[0033] Repeat the above operations until there is no such detection target in the video frame set to mark all center coordinates in the preset blank image.

[0034] Preferably, motion detection is performed on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain the motion trajectory labels of multiple detection targets, and it further includes:

[0035] Calculate the distance between adjacent center coordinates in the preset blank image based on the Euclidean distance formula, and repeat the above operations until all center coordinates in the preset blank image are traversed to obtain the trajectory length of the motion trajectory;

[0036] Calculate the velocity between adjacent central coordinates based on the time interval between adjacent central coordinates in the preset blank image, calculate the acceleration of the motion trajectory based on the velocity between adjacent central coordinates, and repeat the above operations until all central coordinates in the preset blank image are traversed to obtain an acceleration sequence of the motion trajectory;

[0037] Calculate the direction angle between adjacent central coordinates in the preset blank image, and repeat the above operations until all central coordinates in the preset blank image are traversed to obtain a direction angle sequence of the motion trajectory;

[0038] Encode the trajectory length, acceleration sequence, and direction angle sequence of the motion trajectory to obtain a feature vector of the motion trajectory;

[0039] Classify the feature vector of the motion trajectory based on a trained trajectory classification model to obtain a motion trajectory label of the detection target.

[0040] Preferably, the calculation formula for the final static-dynamic state value is as follows:

[0041]

[0042] where s sata represents the final static-dynamic state value, s sata =0 indicates that the final static-dynamic state value is in a static state, s sata =1 indicates that the final static-dynamic state value is in a dynamic state, s represents the first static-dynamic state value, represents the second static-dynamic state value.

[0043] Preferably, the method further includes:

[0044] Construct a monitoring graph structure corresponding to the monitoring area based on the target detection result sets of each video frame in the video frame set, where the monitoring graph structure includes a node set and an edge set, where the nodes in the node set correspond to the monitoring area, and the edges in the edge set correspond to the spatial adjacency relationship, where the spatial adjacency relationship is used to represent that the detection target transfers from one monitoring area to another monitoring area;

[0045] Perform anomaly detection on the monitoring graph structure based on a pre-trained area detection model to obtain an anomaly detection label corresponding to the monitoring area, where the area detection model uses a graph neural network, and the update formula of the graph neural network is as follows:

[0046]

[0047] where, represents the feature vector of node i at the k-th layer, W(k) represents the weight matrix of the k-th layer in the graph neural network, σ represents the activation function, ω ij represents the edge weight between node i and neighbor node j, and θ(i) represents the set of neighbor nodes.

[0048] The technical solution adopted to solve the above technical problems is: A monitoring video detection and recognition system based on deep learning, which is applicable to the above-mentioned monitoring video detection and recognition method based on deep learning, and includes:

[0049] A video frame screening unit, which is used to obtain the target monitoring video of the monitoring area based on the monitoring camera device, obtain the video frame sequence of the target monitoring video, and perform target screening on the video frame sequence to obtain a video frame set;

[0050] A target detection unit, which is used to perform target detection on each video frame in the video frame set based on the trained target detection model to obtain the target detection result set of each video frame in the video frame set;

[0051] A motion detection unit, which is used to perform motion detection on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain the motion trajectory labels of multiple detection targets;

[0052] An event description unit, which is used to extract keywords from the target detection result set and the motion trajectory labels of multiple detection targets to obtain a set of prompt words, input the set of prompt words into the trained large language model, and output the event description set of the target monitoring video based on the trained large language model;

[0053] A text generation unit, which is used to obtain the real-time position based on the GPS module integrated in the monitoring camera device, and generate the scene detection text set of the target monitoring video based on the real-time position and the event description set of the target monitoring video.

[0054] The beneficial effects of the present invention are as follows: (1) Through an automated video analysis process, the present invention significantly improves the working efficiency of video surveillance. After adopting the object detection model and motion analysis technology, video frames can be processed in real time, automatically identifying and tracking targets. This process reduces the input of human resources, making surveillance more efficient. Moreover, through the trained object detection model, various targets can be accurately identified in video frames and detailed behavior analysis can be carried out; (2) Through motion detection technology, the present invention can track the trajectory of targets, providing data support for identifying suspicious behaviors. This accuracy is particularly important in fields such as public safety and traffic management, effectively preventing and responding to potential security threats. Moreover, through the combination of keyword extraction and large language models, complex surveillance data can be converted into structured event descriptions. Such textual information not only facilitates managers to quickly understand the on-site situation but also provides a reference for subsequent event processing. Combining the real-time generated event descriptions, managers can formulate response strategies more efficiently; (3) The present invention integrates the function of the GPS module, which can obtain real-time positions and combine them with surveillance events to generate scene detection texts. This integration of spatial information provides a more comprehensive background for event analysis, helping to enhance the understanding of the on-site situation and improve the accuracy of event processing. Moreover, by constructing a surveillance graph structure and applying graph neural networks (GNNs) for anomaly detection, it is possible to more efficiently identify and track the transfer of targets between different surveillance areas. This method can capture the spatial relationships of targets and detect abnormal behaviors based on the update of the surveillance graph structure, thereby improving the accuracy and response speed of the surveillance system. Combining the advantages of object detection results and graph neural networks, automated anomaly detection in surveillance areas is achieved without manual intervention and can be deployed on a large scale in different scenarios, enhancing the intelligence and efficiency of the surveillance system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic flowchart of the steps of the overall method in an embodiment proposed by the present invention;

[0056] Figure 2 It is a schematic diagram of the system architecture of the overall system in an embodiment proposed by the present invention.

[0057] Reference numerals: 1, video frame screening unit; 2, object detection unit; 3, motion detection unit; 4, event description unit; 5, text generation unit. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] Embodiment 1, as Figure 1 shown, the method for detecting and identifying surveillance videos based on deep learning proposed by the present invention includes:

[0059] ]S1. Obtain the target surveillance video of the surveillance area based on the surveillance camera device, obtain the video frame sequence of the target surveillance video, and perform target screening on the video frame sequence to obtain a video frame set;

[0060] S2. Perform target detection on each video frame in the video frame set based on the pre-trained target detection model to obtain the target detection result set of each video frame in the video frame set;

[0061] S3. Perform motion detection on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain the motion trajectory labels of multiple detection targets;

[0062] S4. Extract keywords from the target detection result set and the motion trajectory labels of multiple detection targets to obtain a set of prompt words, input the set of prompt words into the pre-trained large language model, and output the event description set of the target surveillance video based on the pre-trained large language model;

[0063] S5. Obtain the real-time position based on the GPS module integrated in the surveillance camera device, and generate the scene detection text set of the target surveillance video based on the real-time position and the event description set of the target surveillance video.

[0064] In the present invention, the surveillance camera device is used to capture the device in the surveillance area in real time; the video frame sequence means that the surveillance video is composed of a series of consecutive static images (frames), and the video frame sequence refers to the set of these static images; target screening means extracting specific objects (such as pedestrians, vehicles, etc.) from the video frames to generate a video frame set; keyword extraction means extracting important words or phrases from the target detection results and motion trajectory labels, usually using natural language processing techniques; the large language model (LLM) is a deep learning model for generating text, and generates relevant event descriptions based on the set of prompt words; the GPS module is a global positioning system module integrated in the surveillance device, used to obtain the real-time position information of the device; the scene detection text set refers to the text generated based on the real-time position and event description, usually including specific descriptions of the scene, location names, etc.

[0065] In the second embodiment, the surveillance video detection and recognition method based on deep learning proposed by the present invention, compared with the first embodiment, this embodiment further includes: the target detection result set includes the target detection box coordinates, the target detection result confidence, the target category, and the target appearance feature information, where the target category includes objects, people, and the environment.

[0066] In an optional embodiment, performing target screening on the video frame sequence to obtain a video frame set includes:

[0067] A1. Traverse the video frame sequence, model the background of the initial video frame in the video frame sequence based on the Gaussian distribution to obtain the background model corresponding to the initial video frame;

[0068] A2. When the current video frame is traversed, use the background model to compare each pixel point of the current video frame one by one to classify the pixel points of the current video frame and determine the type of the pixel points. Among them, the types of pixel points include background points and foreground points;

[0069] A3. Count the number of foreground points determined in the current video frame to obtain the number of foreground points, and calculate the retention coefficient of the current video frame based on the number of foreground points to obtain the retention coefficient of the current video frame;

[0070] A4. Compare the retention coefficient of the current video frame with the preset retention threshold. If the retention coefficient of the current video frame is greater than the preset retention threshold, add the current video frame to the video frame set;

[0071] A5. Update the background model based on the current video frame to obtain a new background model, and repeat the above operations until all video frames in the video frame sequence are traversed to obtain the video frame set corresponding to the video frame sequence.

[0072] It should be noted that the Gaussian distribution (or normal distribution) is a common probability distribution used to describe many natural phenomena. In video processing, the Gaussian distribution can be used to model the color value distribution of background pixels to facilitate the distinction between the background and the foreground; the background model is a mathematical representation of the invariant background in the video, used to determine which pixels are the background (the static part) and which are the foreground (the dynamic part), and it is constructed by statistically analyzing the pixel values in multiple frames of images; the retention coefficient is an index used to quantify the importance of the current video frame, usually calculated based on the number of foreground points in the current frame. The higher the retention coefficient, the more important the information contained in the current frame; after each frame is processed, the background model update refers to adjusting the background model according to the classification results of the foreground and background in the current frame to make it more accurately reflect the changes in the current environment. This step is to improve the processing accuracy of subsequent frames.

[0073] In an optional embodiment, the background model is as follows:

[0074]

[0075] where x j,t represents the pixel value of the j-th pixel point at time t in the video frame, P represents the background distribution of the pixel point, represents the weight value of the i-th Gaussian distribution at time t in the background model, denotes the mean value of the i-th Gaussian distribution of the j-th pixel at time t in the video frame, denotes the covariance matrix of the i-th Gaussian distribution of the j-th pixel at time t in the video frame, where, and denotes the average pixel values of the R, G, and B components of the j-th pixel at time t in the video frame in the RGB color space, where, and denotes the standard deviation of the pixel values of the R, G, and B components of the j-th pixel at time t in the video frame in the RGB color space, η denotes the probability density function of the Gaussian distribution, where,

[0076] The calculation formula of the retention coefficient is as follows:

[0077]

[0078] where, τ denotes the retention coefficient, q denotes the number of foreground pixels in the video frame, and m*n denotes the total number of pixels in the video frame;

[0079] The update formula of the background model is as follows:

[0080]

[0081] where, M i,t denotes whether the pixel is a foreground point, M i,t =1 indicates that the pixel is a foreground point, M i,t =0 indicates that the pixel is a background point, X t denotes the pixel value of the pixel, and α and ρ denote preset weight coefficients.

[0082] In an optional embodiment, the target detection model extracts features of different scales of the video frame through 3 efficient convolution modules, which are respectively denoted as C1, C2, and C3. Multi-scale feature fusion is used to perform target recognition on features of different scales. Among them, for the smallest-scale feature C3, first, a 1×1 convolution operation is performed on the smallest-scale feature C3 for feature transformation, and then an upsampling operation is performed to obtain a feature of the same size as C2. Then, this feature is fused with C2 and input into an efficient convolution module to extract the fused feature. Next, the fused feature is mapped to the same size as C1 through an upsampling operation, and it is concatenated with C1 at the channel level. Finally, an efficient convolution module is used to extract the final feature, and this feature is input into a classification module to obtain the target detection result set of the video frame.

[0083] It should be noted that the efficient convolution module refers to an optimized structure adopted in a convolutional neural network (CNN), aiming to improve the efficiency and effect of convolutional operations. These modules usually contain various convolutional operations, such as depthwise separable convolution, which can maintain the feature extraction ability while reducing the computational amount; multi-scale feature fusion combines features from different scales (sizes) to improve the accuracy of object detection, which means the model can better identify objects of various sizes because features of different scales provide different levels of information; upsampling is an operation to increase the size of the feature map, usually used to restore the spatial information of the feature map.

[0084] In an optional embodiment, the efficient convolution module uses two different branches to process the input features. Among them, the first branch first extracts the input channel-dependent features through a 3×3 depthwise convolution kernel, and then extracts the input spatial-dependent features through a 1×1 pixelwise convolution operation. Among them, the second branch first extracts the input spatial-dependent features through a 1×1 pixelwise convolution operation, then uses a 3×3 depth convolution operation to extract the channel-dependent features of the input features, and then weights each channel through an attention module to extract the weighted features. Finally, a 1×1 pixelwise convolution operation is used to extract the spatial-dependent features of the weighted features. The features of the two branches are integrated through channel-level concatenation, and the features of the two branches are integrated through a 3×3 depth convolution operation and a 1×1 pixelwise convolution operation.

[0085] It should be noted that the spatial-dependent features refer to the relationship between the features at a certain position and the features at the surrounding positions on the feature map. This dependence relationship can help the model understand the structure and pattern within the local area; channel-level concatenation is to splice the feature maps from different branches in the channel dimension, which can effectively integrate features from different sources and enhance the network's utilization of information.

[0086] In an optional embodiment, motion detection is performed on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain the motion trajectory labels of multiple detection targets, including:

[0087] B1. Traverse the video frame set, compare the coordinates of the target detection boxes in the target detection result set of the current frame with the coordinates of the target detection boxes in the previous frame of the video frame set to obtain the first static / dynamic state value of the detection target in the current frame;

[0088] B2. Based on the target detection result set, obtain the second static / dynamic state value of the detection target in the current frame through Kalman filtering, and obtain the final static / dynamic state value of the detection target in the current frame based on the first static / dynamic state value and the second static / dynamic state value;

[0089] B4. When the final static / dynamic state value of the detected target in the current frame is the dynamic state, obtain the center coordinates of the target detection box of the detected target in the current frame, and mark the center coordinates in a preset blank image;

[0090] B5. Repeat the above operations until there are no detected targets in the video frame set, so as to mark all the center coordinates in the preset blank image.

[0091] It should be noted that Kalman filtering is a recursive algorithm commonly used to estimate the state of a dynamic system. In target tracking, Kalman filtering can be used to predict the next position of the target and at the same time fuse the current observation information to reduce the influence of noise.

[0092] In an optional embodiment, based on the target detection result sets of each video frame in the video frame set, perform motion detection on the detected targets in the video frame set to obtain the motion trajectory labels of multiple detected targets, and further include:

[0093] B6. Calculate the distance between adjacent center coordinates in the preset blank image based on the Euclidean distance formula, and repeat the above operations until all the center coordinates in the preset blank image are traversed to obtain the trajectory length of the motion trajectory;

[0094] B7. Calculate the speed between adjacent center coordinates based on the time interval between adjacent center coordinates in the preset blank image, calculate the acceleration of the motion trajectory based on the speed between adjacent center coordinates, and repeat the above operations until all the center coordinates in the preset blank image are traversed to obtain the acceleration sequence of the motion trajectory;

[0095] B8. Calculate the direction angle between adjacent center coordinates in the preset blank image, and repeat the above operations until all the center coordinates in the preset blank image are traversed to obtain the direction angle sequence of the motion trajectory;

[0096] B9. Encode the trajectory length, acceleration sequence and direction angle sequence of the motion trajectory to obtain the feature vector of the motion trajectory;

[0097] B10. Classify the feature vector of the motion trajectory based on the trained trajectory classification model to obtain the motion trajectory label of the detected target.

[0098] It should be noted that the trajectory classification model uses a support vector machine; the motion trajectory labels include abnormal labels and normal labels.

[0099] In an optional embodiment, the calculation formula of the final static / dynamic state value is as follows:

[0100]

[0101] Among them, s sata represents the final static-dynamic state value. s sata = 0 indicates that the final static-dynamic state value is the static state. s sata = 1 indicates that the final static-dynamic state value is the dynamic state. s represents the first static-dynamic state value, represents the second static-dynamic state value.

[0102] It should be noted that each frame in the video frame set is traversed, and the target detection box of the current frame is compared with that of the previous frame. By comparing the positions of the target detection boxes (usually the center coordinates of the detection boxes), the state (dynamic or static) of the target in the current frame compared with the previous frame can be obtained. If the position of the target detection box has changed significantly, it is determined as a dynamic target; if the detection box has hardly changed, it is determined as a static target. The Kalman filter is a dynamic system estimation method based on statistics and is usually used to estimate the state of an object in a noisy environment. In target detection, the Kalman filter is usually used to predict the target position and predict the position of the current frame based on the state of the previous frame. Based on the result of the Kalman filter, the Kalman filter will predict the target position in the current frame according to the motion trend of the target. If there is a significant difference between the predicted position of the Kalman filter and the actually detected position, this may indicate that the target is moving. If the difference between the two is very small, it can be considered that the target is in a stationary state. The final static-dynamic state value is determined by combining the first static-dynamic state value and the second static-dynamic state value. Usually, if the first static-dynamic state value and the second static-dynamic state value are the same (for example, both are expressed as "dynamic" or "static"), the final static-dynamic state value takes the same state.

[0103] In an optional embodiment, the method further includes:

[0104] Construct a monitoring graph structure corresponding to the monitoring area based on the target detection result set of each video frame in the video frame set. Among them, the monitoring graph structure includes a node set and an edge set. Among them, the nodes in the node set correspond to the monitoring area, and the edges in the edge set correspond to the spatial adjacency relationship. Among them, the spatial adjacency relationship is used to represent that the detected target transfers from one monitoring area to another monitoring area;

[0105] Perform anomaly detection on the monitoring graph structure based on a pre-trained region detection model to obtain an anomaly detection label corresponding to the monitoring area. Among them, the region detection model uses a graph neural network, and the update formula of the graph neural network is as follows:

[0106]

[0107] Among them, represents the feature vector of node i at the k-th layer, W (k)denotes the weight matrix of the k-th layer in the graph neural network, σ denotes the activation function, ω ij denotes the edge weight between node i and neighbor node j, and θ(i) denotes the set of neighbor nodes.

[0108] It should be noted that the Graph Neural Network (GNN) is a deep learning model specialized for processing graph-structured data. Different from traditional neural networks, GNN can capture the relationship information between nodes in graph-structured data and update the features of each node through a message passing mechanism.

[0109] Example 3, as Figure 2 shown, the monitoring video detection and recognition system based on deep learning proposed by the present invention, which is applicable to the monitoring video detection and recognition method based on deep learning, includes:

[0110] Video frame screening unit 1, which is used to obtain the target monitoring video of the monitoring area based on the monitoring camera device, obtain the video frame sequence of the target monitoring video, and perform target screening on the video frame sequence to obtain a video frame set;

[0111] Target detection unit 2, which is used to perform target detection on each video frame in the video frame set based on the trained target detection model to obtain the target detection result set of each video frame in the video frame set;

[0112] Motion detection unit 3, which is used to perform motion detection on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain the motion trajectory labels of multiple detection targets;

[0113] Event description unit 4, which is used to extract keywords from the target detection result set and the motion trajectory labels of multiple detection targets to obtain a set of prompt words, input the set of prompt words into the trained large language model, and output the event description set of the target monitoring video based on the trained large language model;

[0114] Text generation unit 5, which is used to obtain the real-time position based on the GPS module integrated in the monitoring camera device, and generate the scene detection text set of the target monitoring video based on the real-time position and the event description set of the target monitoring video.

[0115] The above has described the embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made without departing from the spirit of the present invention within the knowledge scope of those skilled in the art.

Claims

1. A monitoring video detection and recognition method based on deep learning, characterized in that Including: Based on a monitoring camera device, obtain a target monitoring video of a monitoring area, obtain a video frame sequence of the target monitoring video, and perform target screening on the video frame sequence to obtain a video frame set; Based on a trained target detection model, perform target detection on each video frame in the video frame set to obtain a target detection result set for each video frame in the video frame set; Based on the target detection result set of each video frame in the video frame set, perform motion detection on the detection targets in the video frame set to obtain motion trajectory labels of multiple detection targets; Extract keywords from the target detection result set and the motion trajectory labels of the multiple detection targets to obtain a set of prompt words, input the set of prompt words into a trained large language model, and based on the trained large language model, output an event description set of the target monitoring video; Based on a GPS module integrated in the monitoring camera device, obtain a real-time position, and generate a scene detection text set of the target monitoring video based on the real-time position and the event description set of the target monitoring video; Based on the target detection result set of each video frame in the video frame set, perform motion detection on the detection targets in the video frame set to obtain motion trajectory labels of multiple detection targets, including: Traverse the video frame set, compare the target detection box coordinates in the target detection result set of the current frame with the target detection box coordinates of the previous frame in the video frame set to obtain a first static / dynamic state value of the detection target in the current frame; Based on the target detection result set, obtain a second static / dynamic state value of the detection target in the current frame based on Kalman filtering, and obtain a final static / dynamic state value of the detection target in the current frame based on the first static / dynamic state value and the second static / dynamic state value; When the final static / dynamic state value of the detection target in the current frame is a dynamic state, obtain the center coordinates of the target detection box of the detection target in the current frame, and mark the center coordinates in a preset blank image; Repeat the above operations until there are no detection targets in the video frame set to mark all center coordinates in the preset blank image; Calculate the distance between adjacent center coordinates in the preset blank image based on the Euclidean distance formula, and repeat the above operations until all center coordinates in the preset blank image are traversed to obtain the trajectory length of the motion trajectory; Calculate the speed between adjacent center coordinates based on the time interval between adjacent center coordinates in the preset blank image, calculate the acceleration of the motion trajectory based on the speed between adjacent center coordinates, and repeat the above operations until all center coordinates in the preset blank image are traversed to obtain an acceleration sequence of the motion trajectory; Calculate the direction angle between adjacent center coordinates in the preset blank image, and repeat the above operations until all center coordinates in the preset blank image are traversed to obtain a direction angle sequence of the motion trajectory; Encode the trajectory length, acceleration sequence, and direction angle sequence of the motion trajectory to obtain the feature vector of the motion trajectory; Classify the feature vector of the motion trajectory based on the trained trajectory classification model to obtain the motion trajectory label of the detection target; The method further includes: Construct a monitoring graph structure corresponding to the monitoring area based on the target detection result sets of each video frame in the video frame set, where the monitoring graph structure includes a node set and an edge set, where the nodes in the node set correspond to the monitoring area, and the edges in the edge set correspond to the spatial adjacency relationship, where the spatial adjacency relationship is used to represent the transfer of the detection target from one monitoring area to another monitoring area; Perform anomaly detection on the monitoring graph structure based on the pre-trained area detection model to obtain the anomaly detection label corresponding to the monitoring area; Perform target screening on the video frame sequence to obtain a video frame set, including: Traverse the video frame sequence, and model the background of the initial video frame in the video frame sequence based on the Gaussian distribution to obtain the background model corresponding to the initial video frame; When traversing to the current video frame, compare the pixel points of the current video frame with the background model one by one to classify the pixel points of the current video frame to determine the type of the pixel points, where the types of the pixel points include background points and foreground points; Count the number of foreground points determined in the current video frame to obtain the number of foreground points, and calculate the retention coefficient of the current video frame based on the number of foreground points to obtain the retention coefficient of the current video frame.

2. The monitoring video detection and recognition method based on deep learning according to claim 1, characterized in that The target detection result set includes target detection box coordinates, target detection result confidence, target category, and target appearance feature information, where the target category includes objects, people, and the environment.

3. The method for monitoring video detection and recognition based on deep learning according to claim 2, wherein Performing target screening on the video frame sequence to obtain a video frame set further includes: Compare the retention coefficient of the current video frame with a preset retention threshold. If the retention coefficient of the current video frame is greater than the preset retention threshold, add the current video frame to the video frame set; Update the background model based on the current video frame to obtain a new background model, and repeat the above operations until all video frames in the video frame sequence are traversed to obtain the video frame set corresponding to the video frame sequence.

4. The method for detecting and identifying surveillance videos based on deep learning according to claim 3, characterized in that, The background model is as follows: where x j,t represents the pixel value of the j-th pixel at time t in the video frame, P represents the background distribution of the pixel, represents the weight value of the i-th Gaussian distribution at time t in the background model, represents the mean value of the i-th Gaussian distribution of the j-th pixel at time t in the video frame, represents the covariance matrix of the i-th Gaussian distribution of the j-th pixel at time t in the video frame, where, and represent the average pixel values of the R, G, and B components of the j-th pixel at time t in the video frame in the RGB color space, where, and represent the standard deviations of the pixel values of the R, G, and B components of the j-th pixel at time t in the video frame in the RGB color space, η represents the probability density function of the Gaussian distribution, where, The calculation formula of the retention coefficient is as follows: where τ represents the retention coefficient, q represents the number of foreground pixel points in the video frame, and m*n represents the total number of pixel points in the video frame; The update formula of the background model is as follows: Among them, M i,t indicates whether the pixel is a foreground point. M i,t = 1 indicates that the pixel is a foreground point. M i,t = 0 indicates that the pixel is a background point. X t represents the pixel value of the pixel, and α and ρ represent preset weight coefficients.

5. The method for detecting and identifying surveillance videos based on deep learning according to claim 2, wherein, The target detection model extracts features of different scales of the video frame through 3 efficient convolution modules, denoted as C1, C2, and C3 respectively, and uses multi-scale feature fusion to perform target recognition on features of different scales. Among them, for the smallest-scale feature C3, first, a 1×1 convolution operation is used to perform feature transformation on the smallest-scale feature C3, and then an upsampling operation is used to obtain a feature of the same size as C2. Then, this feature is fused with C2 and input into an efficient convolution module to extract the fused feature. Next, the fused feature is mapped to the same size as C1 through an upsampling operation and concatenated with C1 at the channel level. Finally, an efficient convolution module is used to extract the final feature, and this feature is input into a classification module to obtain the target detection result set of the video frame.

6. The method for detecting and recognizing surveillance videos based on deep learning according to claim 5, wherein The efficient convolution module uses two different branches to process the input feature. Among them, the first branch first extracts the input channel-dependent feature through a 3×3 depthwise convolution kernel, and then extracts the input spatial-dependent feature through a 1×1 pointwise convolution operation. Among them, the second branch first extracts the input spatial-dependent feature through a 1×1 pointwise convolution operation, then uses a 3×3 depth convolution operation to extract the channel-dependent feature of the input feature, and then weights each channel through an attention module to extract the weighted feature. Finally, a 1×1 pointwise convolution operation is used to extract the spatial-dependent feature of the weighted feature. The features of the two branches are integrated through channel-level concatenation, and the features of the two branches are integrated through a 3×3 depth convolution operation and a 1×1 pointwise convolution operation.

7. The method for monitoring video detection and recognition based on deep learning according to claim 1, characterized in that The calculation formula of the final static and dynamic state value is as follows: Among them, s sata represents the final static-dynamic state value. s sata = 0 indicates that the final static-dynamic state value is in the static state. s sata = 1 indicates that the final static-dynamic state value is in the dynamic state. s represents the first static-dynamic state value. represents the second static-dynamic state value.

8. The method for detecting and recognizing surveillance videos based on deep learning according to claim 2, wherein The region detection model uses a graph neural network, and the update formula of the graph neural network is as follows: Among them, represents the feature vector of node i at the k-th layer, W (k) represents the weight matrix of the k-th layer in the graph neural network, σ represents the activation function, ω ij represents the edge weight between node i and neighbor node j, and θ(i) represents the set of neighbor nodes.

9. A monitoring video detection and recognition system based on deep learning, which is applicable to the method for detecting and recognizing monitoring videos based on deep learning according to any one of claims 1-8, characterized in that, Including: A video frame screening unit (1), where the video frame screening unit (1) is used to obtain the target monitoring video of the monitoring area based on the monitoring camera device, obtain the video frame sequence of the target monitoring video, and perform target screening on the video frame sequence to obtain a video frame set; A target detection unit (2), where the target detection unit (2) is used to perform target detection on each video frame in the video frame set based on the trained target detection model to obtain the target detection result set of each video frame in the video frame set; A motion detection unit (3), where the motion detection unit (3) is used to perform motion detection on the detection targets in the video frame set based on the target detection result set of each video frame in the video frame set to obtain the motion trajectory labels of multiple detection targets; An event description unit (4), where the event description unit (4) is used to extract keywords from the target detection result set and the motion trajectory labels of the multiple detection targets to obtain a set of prompt words, input the set of prompt words into the trained large language model, and output the event description set of the target monitoring video based on the trained large language model; A text generation unit (5), which is configured to obtain a real-time location based on a GPS module integrated in the monitoring camera device, and generate a set of scene detection texts for the target monitoring video based on the real-time location and a set of event descriptions of the target monitoring video.

Images