Video analysis method and system for monitoring terminal

Abstract

The present application relates to the technical field of video analysis, and discloses a video analysis method and system for a monitoring terminal, the method comprising: adaptively adjusting the frame rate and performing background modeling on a real-time video stream collected by a monitoring terminal to obtain an adaptive background model and a target detection result containing a monitoring object; performing feature processing on the target detection result based on a multi-scale temporal receptive field network to obtain a multi-scale temporal feature tensor; performing coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature graph, and performing temporal attention allocation on the optimized spatiotemporal feature graph to obtain a weighted key frame feature sequence; adaptively performing feature fusion on the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and performing hierarchical action recognition on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report, thereby improving the accuracy of the monitoring terminal in recognizing the action of the monitoring object.

Description

Technical Field

[0001] The present application relates to the technical field of video analysis, and in particular to a video analysis method and system for a monitoring terminal. Background Art

[0002] With the rapid development of Internet of Things technology and intelligent monitoring systems, video surveillance terminals have been widely used in public safety, traffic management, urban governance and other fields. Real-time analysis and processing of massive video data has become a hot topic in current research. The traditional cloud-centric analysis model faces challenges such as data transmission and real-time performance, and it is difficult to meet the real-time analysis needs in complex scenarios.

[0003] The introduction of edge computing provides a new approach to solving this problem, but there are still many challenges in achieving efficient and accurate video analysis on edge devices with limited computing resources. Existing methods based on the image domain often have high computational complexity and are difficult to run in real time on edge devices; while simplified lightweight models often have difficulty coping with complex and changing actual scenarios. In addition, how to effectively capture action features at different time scales, how to reduce computational complexity while ensuring analysis accuracy, and how to achieve rapid recognition of newly emerging action categories are all issues that need to be addressed urgently. Summary of the invention

[0004] The present application provides a video analysis method and system for a monitoring terminal, thereby improving the accuracy of the monitoring terminal in identifying the actions of a monitored object.

[0005] A first aspect of the present application provides a video analysis method for a monitoring terminal, the video analysis method for a monitoring terminal comprising:

[0006] Adaptively adjust the frame rate and background modeling of the real-time video stream collected by the monitoring terminal to obtain an adaptive background model and target detection results containing the monitored object;

[0007] Performing feature processing on the target detection result based on a multi-scale temporal receptive field network to obtain a multi-scale temporal feature tensor;

[0008] Performing coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map, and performing temporal attention allocation on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence;

[0009] Adaptive feature fusion is performed on the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and hierarchical action recognition is performed on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

[0010] A second aspect of the present application provides a video analysis system for a monitoring terminal, the video analysis system for the monitoring terminal comprising:

[0011] A modeling module is used to perform adaptive frame rate adjustment and background modeling on the real-time video stream collected by the monitoring terminal to obtain an adaptive background model and target detection results including the monitored object;

[0012] A processing module, used for performing feature processing on the target detection result based on a multi-scale time receptive field network to obtain a multi-scale time feature tensor;

[0013] An analysis module, configured to perform coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map, and perform temporal attention allocation on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence;

[0014] The recognition module is used to perform adaptive feature fusion on the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and perform hierarchical action recognition on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

[0015] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned video analysis method of the monitoring terminal.

[0016] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned video analysis method for a monitoring terminal.

[0017] Compared with the prior art, the present application has the following beneficial effects: Adaptive frame rate adjustment and background modeling technology can dynamically adjust the processing strategy according to the scene complexity and terminal computing power, effectively balancing the analysis efficiency and resource consumption. The introduction of the multi-scale time receptive field network enables the system to capture short-term and long-term time features at the same time, improving the recognition ability of actions of different durations. The combination of coordinate attention and pyramid pooling enhances the expression ability of spatial features while reducing the interference of background noise. The nonlinear kernel function density peak clustering and time attention allocation mechanism can effectively extract key frames and assign different importance to different frames, thereby improving the efficiency of feature representation. Graph convolution operation and adaptive feature fusion technology make full use of the complementarity of spatiotemporal information and enhance the discriminative ability of features. The hierarchical action recognition strategy combines meta-learning and Bayesian analysis, which not only improves the accuracy of action recognition of the monitoring terminal for the monitored object, but also enhances the model's adaptability to new action categories. This method adopts a series connection of multiple lightweight modules to reduce the computational complexity while ensuring the analysis performance, and is suitable for deployment on edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0020] Figure 1 1 is a flow chart of a video analysis method for a monitoring terminal provided by an embodiment of the present invention;

[0021] Figure 2 It is a schematic block diagram of the structure of a video analysis system for a monitoring terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 , an embodiment of the video analysis method of the monitoring terminal in the embodiment of the present application includes:

[0026] Step 100: adaptively adjust the frame rate and perform background modeling on the real-time video stream collected by the monitoring terminal to obtain an adaptive background model and a target detection result including the monitored object;

[0027] It is understandable that the execution subject of the present application may be a video analysis system of a monitoring terminal, or may be a terminal or a server, which is not specifically limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0028] Specifically, the scene complexity of the real-time video stream collected by the monitoring terminal is evaluated, and the scene complexity is determined by analyzing factors such as the dynamic and static nature of the video content, the number of objects and their movement speed, and the scene complexity index is obtained. According to the scene complexity index and the computing power of the monitoring terminal, the frame rate of the real-time video stream is dynamically adjusted. Based on the complexity of the current scene and the availability of system resources, the frame rate is appropriately reduced in complex scenes, thereby reducing the processing burden and obtaining an adjusted video stream. Gaussian mixture modeling is performed on the adjusted video stream to establish an initial background model. In this model, the background color and brightness of each pixel are regarded as Gaussian distributions, and the background of the scene is described by a linear combination of multiple Gaussian models. Inter-frame difference processing is performed on the initial background model and the adjusted video stream, and by comparing the difference between the current frame and the background model, possible motion areas are identified to form a motion area mask. The motion area mask is used to identify all areas in the video stream where changes may occur, that is, areas where monitored objects may appear. The initial background model is updated according to the motion area mask information. The update of the background model is an adaptive process, which allows the model to gradually adapt to changes in the scene and obtain a more accurate adaptive background model. The adaptive background model is used to detect the target in the adjusted video stream. By comparing the difference between the current frame and the background model, the possible moving objects in the scene are preliminarily identified to obtain the initial detection results. The initial detection results are analyzed by constructing a region proposal network. The region proposal network is a deep learning technology used to generate candidate regions from an image, which may contain targets of interest. The network analyzes the initial detection results and generates a set of candidate regions, which represent the locations where the monitored objects may exist in the video stream. Feature analysis is performed on the candidate region set, and the specific features in each region are identified through feature extraction technology to obtain the target focus area. Based on the information of the target focus area, the initial detection results are processed for cross-frame target tracking. By tracking the position changes of the target in multiple consecutive frames, the precise position and trajectory of the monitored object are determined, and the target detection result containing the monitored object is obtained.

[0029] Step 200: performing feature processing on the target detection result based on the multi-scale time receptive field network to obtain a multi-scale time feature tensor;

[0030] Specifically, based on the multi-scale temporal receptive field network and target detection results, the real-time video stream is divided into time windows, and the video data is divided into short-term time window sequences and long-term time window sequences. The short-term time window sequence is used to capture subtle changes in a short period of time, while the long-term time window sequence focuses on the overall trend in a longer time span. The short-term time window sequence is processed by a 3D convolutional neural network to extract short-term temporal features, revealing the action or change features in a short period of time and providing more fine-grained time information. At the same time, the long-term time window sequence is processed by an expanded 3D convolutional network to obtain long-term temporal features. The design of the expanded 3D convolutional network enables it to expand the receptive field and more effectively capture information in a long time span. An adaptive fusion module is constructed based on the short-term temporal features and the long-term temporal features. This module is responsible for calculating the fusion weight coefficients, which reflect the relative importance of short-term and long-term features at different time points. Using the weight coefficients, the short-term temporal features and the long-term temporal features are weightedly fused to generate the initial multi-scale temporal features. The initial multi-scale temporal features are dynamically frame sampled according to the activity level of the scene. In high-activity scenes, more frames will be retained, while in low-activity scenes, the sampling frequency can be reduced to optimize the use of computing resources. Motion vector analysis is performed on adjacent frames, which helps to locate high-activity areas, that is, parts of the scene that change dramatically. Based on the high-activity area positioning information obtained, the sampled temporal features are regionally weighted to ensure that the features of the high-activity areas occupy a larger weight in the final feature tensor, thereby highlighting the dynamic changes of these areas. The feature channels of the regional weighted temporal features are reorganized to reorganize the features from different time windows and different receptive fields to generate a unified multi-scale temporal feature tensor.

[0031] Step 300: performing coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map, and performing temporal attention allocation on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence;

[0032] It should be noted that the multi-scale temporal feature tensor is grid segmented to divide the entire feature tensor into multiple feature grid unit sets. Each unit in the feature grid unit set represents the local features of the video in different time and space ranges. The local attention weight is calculated for the feature grid unit, and the local attention map is generated by analyzing the importance of each unit in the overall feature. The multi-scale temporal feature tensor is subjected to global feature extraction, and the global feature vector is generated by integrating the information in the entire feature tensor. According to the local attention map and the global feature vector, the global and local attention weights are analyzed to obtain the coordinate attention matrix, and the weights of the local and global features are integrated to reflect the importance of each feature unit in the overall feature space. The multi-scale temporal feature tensor and the coordinate attention matrix are subjected to attention enhancement processing to enhance the weights of important features in the feature tensor and generate an attention enhancement feature map. The attention enhancement feature map is subjected to multi-branch parallel pooling operation, and features are extracted at different scales through multiple parallel pooling branches to obtain a multi-scale spatial feature set. The multi-scale spatial feature set is subjected to feature splicing, and the output features of different pooling branches are combined to form a spliced ​​feature map. The concatenated feature map is subjected to contrastive learning loss function operation, and the discriminability of the feature is enhanced by contrastive learning method to obtain discriminative enhanced features, ensuring that the features have high distinguishing ability in different situations. According to the complexity of the current scene, the discriminative enhanced features are dynamically dimensionalized to adaptively adjust the dimension of the feature to obtain an adaptive feature representation. The adaptive feature representation is residually connected with the multi-scale temporal feature tensor, and the enhanced features are added while retaining the original features to obtain an optimized spatiotemporal feature map. The optimized spatiotemporal feature map is subjected to nonlinear kernel function density peak clustering to identify the density peak area in the feature map, and temporal attention is allocated to obtain a weighted key frame feature sequence.

[0033] The radial basis function kernel is calculated for the optimized spatiotemporal feature map. By mapping the feature map into the nonlinear feature space, the separability of the features is increased, and a nonlinear feature space mapping is obtained. The local density and relative distance calculations are performed on the nonlinear feature space mapping to measure the density of the feature points in the nonlinear space and the relative distance to other points, and a density-distance matrix is ​​generated. The density peak points are identified according to the density-distance matrix. The peak points represent the high-density areas in the feature space, and the initial cluster center set is determined by these areas. The adaptive bandwidth selection is performed on the initial cluster center set. The location and number of cluster centers are optimized by adjusting the bandwidth parameters to obtain the optimized cluster centers. The optimized cluster centers reflect the main structures and patterns in the feature map. The optimized spatiotemporal feature map is clustered according to the optimized cluster centers, and the feature points in the feature map are assigned to different clusters to form a key frame candidate set. The scene complexity evaluation is performed on the key frame candidate set. The dynamic threshold is calculated by evaluating the dynamics and complexity of the scene. The key frame candidate set is screened according to the dynamic threshold, and those frames that are representative in complex scenes are retained to obtain a key frame set. The key frame set is encoded with temporal relations. By analyzing the temporal order and relations of the key frames, temporal relations encoding is generated to capture the temporal dependency and order information between the key frames. The temporal relations encoding is analyzed with a multi-head self-attention mechanism. The importance of each key frame in the entire sequence is calculated through the self-attention mechanism to obtain the temporal attention weight. The key frame set and the temporal attention weight are weighted and the temporal attention weight is applied to the features of each key frame in the key frame set. The contribution of each key frame is adjusted according to the weight to obtain a weighted key frame feature sequence.

[0034] Step 400: Adaptively fuse the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and perform hierarchical action recognition on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

[0035] Specifically, an initial graph structure is constructed based on the weighted keyframe feature sequence. The keyframe features are represented as graph nodes, and the edge weights are calculated based on the spatiotemporal distance between nodes and feature similarity to generate an edge weight matrix. The edge weight matrix is ​​normalized to standardize the edge weights and obtain a standardized adjacency matrix. Multi-layer graph convolution operations are performed on the weighted keyframe feature sequence according to the standardized adjacency matrix. The local and global spatiotemporal information are fused through the graph convolution network to obtain graph convolution features. The graph convolution features are subjected to residual connection and layer normalization. The enhanced graph features are obtained by retaining the original feature information and standardizing the feature distribution. An adaptive feature fuser is constructed according to the current task type to generate dynamic gating parameters. These parameters are used to control the weights of different features in the fusion process. Task-related features are obtained by performing feature operations on the enhanced graph features and dynamic gating parameters. At the same time, the task-related features and the weighted keyframe feature sequence are decoupled through feature decoupling operations to separate task-related and task-irrelevant features. The task-related features and task-irrelevant features are weightedly fused, and the advantages of the two types of features are combined through a weighted mechanism to obtain preliminary fused features. The initial fusion features are recalibrated and the channel scale is adjusted. By recalibrating the importance of the features and adjusting the channel scale, the fused spatiotemporal feature representation is obtained. Hierarchical action recognition is performed on the fused spatiotemporal feature representation. By refining the features layer by layer, the action features of the monitored object in different time scales and spatial ranges are identified, and a comprehensive video analysis report of the monitored object is generated.

[0036] Convolutional feature operations are performed on the fused spatiotemporal feature representation, and high-level features are extracted by applying convolutional neural networks to obtain a coarse-grained action category probability distribution. The coarse-grained action category probability distribution provides a preliminary action classification result. According to the probability distribution, the corresponding fine-grained classifier is selected to ensure the accuracy and refinement of action recognition. The action category is predicted by the fine-grained classifier on the fused spatiotemporal feature representation to generate a fine-grained action category prediction result. The fine-grained classifier can more accurately identify specific action categories, making the action recognition results more detailed and precise. The fine-grained action category prediction results are matched according to the predefined action prototype library, and the standardized action description is obtained by comparing with the standardized action prototype. The standardized action description is meta-learned to construct the target action recognition model. The meta-learning method improves the adaptability of the model in new tasks by sharing knowledge between different tasks, and enhances the generalization performance of the action recognition model. The target action recognition model is used to identify the newly emerged action categories and generate updated action recognition results. The identification of new action categories ensures that the model can adapt to changing scenarios and needs. The updated action recognition results are analyzed by Bayesian method, and the confidence of action prediction is estimated by Bayesian method. The confidence estimation provides a credibility score for each recognition result, which helps to evaluate the reliability of the recognition result. Based on the confidence estimation of action prediction, the updated action recognition results are subjected to class activation mapping to generate a visualization dataset of action recognition. Class activation mapping makes the action recognition results more intuitive and easy to understand by highlighting the important areas in the feature map. The updated action recognition results, the confidence estimation of action prediction and the visualization dataset of action recognition are combined to generate a comprehensive video analysis report of the monitored object. The report provides detailed action recognition information, including action category, recognition confidence and visualization data, to help users fully understand the behavior and dynamics of the monitored object.

[0037] In the embodiment of the present application, the adaptive frame rate adjustment and background modeling technology can dynamically adjust the processing strategy according to the scene complexity and terminal computing power, effectively balancing the analysis efficiency and resource consumption. The introduction of the multi-scale time receptive field network enables the system to capture short-term and long-term time features at the same time, and improves the recognition ability of actions of different durations. The combination of coordinate attention and pyramid pooling enhances the expression ability of spatial features while reducing the interference of background noise. The nonlinear kernel function density peak clustering and time attention allocation mechanism can effectively extract key frames and assign different importance to different frames, thereby improving the efficiency of feature representation. Graph convolution operations and adaptive feature fusion technology make full use of the complementarity of spatiotemporal information and enhance the discriminative ability of features. The hierarchical action recognition strategy combines meta-learning and Bayesian analysis, which not only improves the accuracy of action recognition of the monitoring terminal for the monitored object, but also enhances the model's adaptability to new action categories. This method adopts a series connection of multiple lightweight modules to reduce the computational complexity while ensuring analysis performance, and is suitable for deployment on edge devices.

[0038] In a specific embodiment, the process of executing step 100 may specifically include the following steps:

[0039] Perform scene complexity evaluation on the real-time video stream collected by the monitoring terminal to obtain a scene complexity index, and dynamically adjust the frame rate of the real-time video stream according to the scene complexity index and the computing power of the monitoring terminal to obtain an adjusted video stream;

[0040] Gaussian mixture modeling is performed on the adjusted video stream to obtain an initial background model, and inter-frame difference processing is performed on the initial background model and the adjusted video stream to obtain a motion region mask;

[0041] The initial background model is updated according to the motion region mask to obtain an adaptive background model;

[0042] The adjusted video stream is subjected to target detection through an adaptive background model to obtain an initial detection result, and a region proposal network is constructed based on the initial detection result to obtain a set of candidate regions;

[0043] The candidate region set is subjected to feature analysis to obtain the target focus region, and the initial detection result is subjected to cross-frame target tracking processing according to the target focus region to obtain the target detection result including the monitored object.

[0044] Specifically, the scene complexity evaluation of the real-time video stream collected by the monitoring terminal is completed by analyzing factors such as the degree of motion in the video frame, lighting changes, and the number and movement speed of objects in the scene. The scene complexity index can be expressed as: ;

[0045] in, represents the scene complexity index, Indicates the degree of exercise, Indicates changes in lighting. Indicates the number of objects, , and is the corresponding weight coefficient. By calculating these indicators, a comprehensive scene complexity score is obtained. According to the scene complexity index and the computing power of the monitoring terminal, the frame rate of the real-time video stream is dynamically adjusted. When the scene complexity is high, the frame rate can be reduced to reduce the computing burden; conversely, when the scene complexity is low, the frame rate can be increased to obtain more detailed information. The adjusted frame rate It can be expressed as: ;

[0046] in, is the maximum frame rate, is the adjustment factor, It is an indicator of scene complexity. Through adjustment, the video details are preserved as much as possible while ensuring computational efficiency. Gaussian mixture modeling is performed on the adjusted video stream. Gaussian mixture model is a commonly used background modeling method that describes the background of each pixel through multiple Gaussian distributions. For each pixel , and its background model can be expressed as ;

[0047] in, is the number of Gaussian components, It is The weights of the Gaussian components, The mean is and the covariance is Gaussian distribution. The initial background model is obtained by training the Gaussian mixture model. The initial background model and the adjusted video stream are subjected to inter-frame difference processing, and the motion region mask is generated by comparing the difference between the current frame and the background model. For each pixel, if the difference between its current value and the background model exceeds a certain threshold, it is marked as a motion region. The generation of the motion region mask can be expressed as:

[0048] ;

[0049] in, represents the motion region mask, is the threshold value, is the mean of the background model. According to the motion region mask, the initial background model is updated to obtain an adaptive background model. By adjusting the parameters of the Gaussian model after each frame, it gradually adapts to the changes in the scene. The updated background model can more accurately reflect the current scene. Through the adaptive background model, the adjusted video stream is subjected to target detection and possible targets are preliminarily identified. Based on the initial detection results, a region proposal network is constructed to generate a set of candidate regions. The region proposal network is a deep learning technology that can generate candidate regions that may contain targets in an image. The region proposal network scans the image through a sliding window technique and predicts whether each window contains a target and the position offset of the window through a small neural network. Feature analysis is performed on the candidate region set to extract the features of each region, and these features are used to identify the target focus area. Feature analysis usually uses a convolutional neural network to extract high-level features of the region through multi-layer convolution and pooling operations. According to the target focus area, the initial detection result is subjected to cross-frame target tracking processing. Target tracking tracks the movement trajectory of the target by analyzing the position changes of the target in consecutive frames, and finally obtains the target detection result containing the monitored object. Cross-frame target tracking usually uses techniques such as Kalman filtering or particle filtering to accurately track the position of the target through prediction and update steps.

[0050] In a specific embodiment, the process of executing step 200 may specifically include the following steps:

[0051] Based on the multi-scale temporal receptive field network and target detection results, the real-time video stream is divided into time windows to obtain short-term time window sequences and long-term time window sequences;

[0052] The short-term time window sequence is processed by a 3D convolutional neural network to obtain short-term time features, and the long-term time window sequence is processed by an expanded 3D convolutional network to obtain long-term time features;

[0053] An adaptive fusion module is constructed according to the short-term time features and the long-term time features, and a fusion weight coefficient is calculated, and the short-term time features and the long-term time features are weightedly fused according to the fusion weight coefficient to obtain an initial multi-scale time feature;

[0054] Perform dynamic frame sampling on the initial multi-scale temporal features according to the activity level of the scene to obtain the sampled temporal features, and perform motion vector analysis on adjacent frames to obtain the high activity area positioning information;

[0055] The sampled time features are regionally weighted according to the positioning information of the high-activity area to obtain the regional weighted time features, and the feature channels of the regional weighted time features are reorganized to obtain a multi-scale time feature tensor.

[0056] Specifically, based on the multi-scale temporal receptive field network and target detection results, the real-time video stream is divided into time windows, and the video data is processed according to different time scales to capture short-term and long-term temporal features. The short-term time window sequence is used to capture subtle changes in a short period of time, while the long-term time window sequence focuses on the overall trend in a longer time span. The short-term time window sequence is processed by a 3D convolutional neural network to extract short-term temporal features. The 3D convolutional neural network performs convolution operations in both time and space, and can capture dynamic changes and spatial features in the video. Through the multi-layer convolution and pooling operations of the 3D convolutional neural network, high-level short-term temporal features are obtained. At the same time, the long-term time window sequence is processed by the dilated 3D convolutional network to extract long-term temporal features. The dilated convolution introduces holes in the convolution kernel to expand the receptive field, and can capture information in a longer time range without increasing the amount of calculation. By dilating the 3D convolutional network, temporal features reflecting long-term changes are obtained. An adaptive fusion module is constructed based on short-term temporal features and long-term temporal features. Features of different time scales are effectively fused to better capture important information in the video. The fusion weight coefficient is calculated by analyzing the importance of short-term and long-term time features and is expressed as: in, is the fusion weight coefficient, is a short-term feature. is a long-term time characteristic. and is a weight parameter that reflects the importance of short-term and long-term features. Through this formula, the initial multi-scale temporal features are obtained. The initial multi-scale temporal features are dynamically frame sampled according to the activity level of the scene. The activity level of the scene is determined by analyzing indicators such as motion vectors and change frequency in the video frames. In high-activity scenes, a higher frame rate is required to capture details, while in low-activity scenes, the frame rate can be reduced to reduce the amount of calculation. Dynamic frame sampling obtains the sampled temporal features. Motion vector analysis is performed on adjacent frames to obtain the positioning information of high-activity areas. Motion vector analysis is to determine the motion area by calculating the pixel motion between adjacent frames. Commonly used methods include optical flow method and frame difference method. By analyzing these motion vectors, the areas with frequent activities in the video are determined. According to the positioning information of the high-activity area, the sampled temporal features are subjected to regional weighting processing. Regional weighting processing is to highlight the importance of high-activity areas so that these areas occupy a larger weight in the final feature representation. The feature representation after regional weighting is: in, is the time characteristic after regional weighting, is the regional weighting coefficient, is the temporal feature after sampling. After processing, the regional weighted temporal feature is obtained. The feature channels of the regional weighted temporal feature are reorganized to obtain a multi-scale temporal feature tensor. By reorganizing the feature channels, the features of different time scales can be effectively combined to form a unified feature representation. The multi-scale temporal feature tensor integrates the information of different time scales and different activity areas, and can more comprehensively reflect the dynamic changes in the video.

[0057] In a specific embodiment, the process of executing step 300 may specifically include the following steps:

[0058] The multi-scale temporal feature tensor is grid-segmented to obtain a feature grid unit set, and the local attention weight is calculated for each unit in the feature grid unit set to obtain a local attention map;

[0059] Perform global feature extraction on the multi-scale temporal feature tensor to obtain the global feature vector, and perform global and local attention weight analysis based on the local attention map and the global feature vector to obtain the coordinate attention matrix;

[0060] Perform attention enhancement processing on the multi-scale temporal feature tensor and the coordinate attention matrix to obtain an attention-enhanced feature map, and perform multi-branch parallel pooling operations on the attention-enhanced feature map to obtain a multi-scale spatial feature set;

[0061] Perform feature splicing on the multi-scale spatial feature set to obtain a spliced ​​feature map, and perform contrastive learning loss function operation on the spliced ​​feature map to obtain discriminative enhancement features;

[0062] According to the complexity of the current scene, the discriminative enhancement features are dynamically dimensionalized to obtain an adaptive feature representation, and the adaptive feature representation is residually connected with the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map;

[0063] Nonlinear kernel function density peak clustering and temporal attention allocation are performed on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence.

[0064] Specifically, the multi-scale temporal feature tensor is grid segmented, and the entire feature tensor is divided into multiple feature grid unit sets. By dividing the feature map into small blocks, it helps to capture the information of the local area. The local attention weight is calculated for each unit in the feature grid unit set, and a local attention map is generated by analyzing the importance of each unit in the overall feature. The local attention weight can be calculated by the following formula: in, It is Unit pair The attention weight of each unit, and They are and The query and key representation of each unit. Perform global feature extraction on the multi-scale temporal feature tensor, and obtain the global feature vector by performing global operations on the entire feature tensor. Perform global and local attention weight analysis based on the local attention map and the global feature vector to obtain the coordinate attention matrix. The coordinate attention matrix combines the weights of local and global features to reflect the importance of each feature unit in the overall feature space. Perform attention enhancement processing on the multi-scale temporal feature tensor and the coordinate attention matrix to enhance the weights of important features in the feature tensor and generate an attention-enhanced feature map. Perform multi-branch parallel pooling operations on the attention-enhanced feature map, and extract features at different scales through multiple parallel pooling branches to obtain a multi-scale spatial feature set. Perform feature splicing on the multi-scale spatial feature set, and combine the output features of different pooling branches to form a spliced ​​feature map. Perform contrastive learning loss function operation on the spliced ​​feature map, and enhance the discriminability of the features through contrastive learning methods to obtain discriminative enhanced features. The contrastive learning loss function can be expressed as: ;

[0065] in, is the contrastive learning loss, Indicates and Whether the samples belong to the same class, and It is and The feature representation of a sample is is a threshold. According to the complexity of the current scene, the discriminative enhanced features are dynamically dimensionalized to adaptively adjust the feature dimensions and obtain an adaptive feature representation. The adaptive feature representation is residually connected with the multi-scale temporal feature tensor, and the enhanced features are added while retaining the original features to obtain an optimized spatiotemporal feature map. The optimized spatiotemporal feature map is subjected to nonlinear kernel function density peak clustering, and the density peak area in the feature map is identified by this clustering method. Then, temporal attention allocation is performed to obtain a weighted key frame feature sequence. Temporal attention allocation is to analyze the temporal importance of key frames and perform weighted processing on them, thereby highlighting the importance of key frames in the feature sequence.

[0066] In a specific embodiment, the process of performing nonlinear kernel function density peak clustering and temporal attention allocation on the optimized spatiotemporal feature graph to obtain a weighted key frame feature sequence may specifically include the following steps:

[0067] Perform radial basis function kernel calculation on the optimized spatiotemporal feature map to obtain a nonlinear feature space mapping, and perform local density and relative distance calculation on the nonlinear feature space mapping to obtain a density-distance matrix;

[0068] According to the density-distance matrix, the density peak point is identified to obtain the initial cluster center set, and the adaptive bandwidth selection is performed on the initial cluster center set to obtain the optimized cluster center;

[0069] The optimized spatiotemporal feature graph is clustered and allocated according to the optimized cluster center to obtain a key frame candidate set, and the scene complexity of the key frame candidate set is evaluated to obtain a dynamic threshold;

[0070] The key frame candidate set is screened according to the dynamic threshold to obtain a key frame set, and the key frame set is encoded in a temporal relationship to obtain a temporal relationship encoding;

[0071] A multi-head self-attention mechanism analysis is performed on the temporal relationship encoding to obtain the temporal attention weight, and a weighted analysis is performed on the key frame set and the temporal attention weight to obtain the weighted key frame feature sequence.

[0072] Specifically, the radial basis function (RBF) kernel is calculated on the optimized spatiotemporal feature map to map the data into a high-dimensional nonlinear feature space. The RBF kernel function is a commonly used kernel function and is defined as follows: ;

[0073] in, Represents the input sample and The similarity between is the kernel width parameter. Through the RBF kernel function, the nonlinear feature space mapping is obtained. The local density and relative distance calculations are performed on the nonlinear feature space mapping to obtain the density-distance matrix. Local density By calculating the sample It is expressed as the number of other samples within a certain radius:

[0074] ;

[0075] in, It is a sample and samples The distance between is a cutoff distance, is the indicator function, when hour, is equal to 1, otherwise it is equal to 0. Relative distance Can be represented as a sample Minimum distance to other samples with higher density: According to the local density and relative distance, a density-distance matrix is ​​constructed. The density and relative distance of each sample form a point, so that the density peak point can be identified. According to the density-distance matrix, the density peak point is identified to obtain the initial cluster center set. The density peak points are those samples with high density and large relative distance, which are used as the initial cluster centers. Adaptive bandwidth selection is performed on the initial cluster center set. The bandwidth parameter can be selected by the optimal bandwidth selection method, such as Silverman's Rule of Thumb, to ensure that the cluster center can effectively cover the data space. According to the optimized cluster center, the optimized spatiotemporal feature map is clustered and assigned, and each sample is assigned to the nearest cluster center to form a key frame candidate set. The scene complexity of the key frame candidate set is evaluated, and the degree of dynamic change of the scene is calculated to obtain a dynamic threshold. The scene complexity evaluation can be completed by calculating the difference between key frames, the change of motion vectors and other indicators. According to the dynamic threshold, the key frame candidate set is screened, and those frames that are representative in complex scenes are retained to obtain a key frame set. Temporal relationship encoding is performed on the key frame set to capture the temporal dependency between key frames and form a temporal relationship encoding. Temporal relationship encoding can be achieved by calculating the time interval, sequence and other information between key frames. The multi-head self-attention mechanism is analyzed for temporal relationship encoding, and the importance of each key frame in the entire sequence is calculated to obtain the temporal attention weight. The multi-head self-attention mechanism is a technology to enhance the attention of the model. It captures the characteristics of different subspaces through multiple attention heads. The temporal attention weight is calculated by the following formula:

[0076] ;

[0077] in, Indicates Keyframe pair The attention weight of key frames, and They are and The query and key representation of key frames. The key frame set and the temporal attention weight are weighted analysis, the temporal attention weight is applied to the features of each key frame in the key frame set, the contribution of each key frame is adjusted according to the weight, and the weighted key frame feature sequence is obtained.

[0078] In a specific embodiment, the process of executing step 400 may specifically include the following steps:

[0079] Construct an initial graph structure based on the weighted key frame feature sequence, calculate the spatiotemporal distance and feature similarity between nodes based on the initial graph structure, and generate an edge weight matrix;

[0080] Normalize the edge weight matrix to obtain a standardized adjacency matrix, and perform multi-layer graph convolution operations on the weighted key frame feature sequence according to the standardized adjacency matrix to obtain graph convolution features;

[0081] Perform residual connection and layer normalization processing on the graph convolution features to obtain enhanced graph features, and build an adaptive feature fuser according to the current task type to obtain dynamic gating parameters;

[0082] Perform feature operations on the enhanced graph features and dynamic gating parameters to obtain task-related features, and perform feature decoupling on the task-related features and weighted keyframe feature sequences to obtain task-related and task-independent features;

[0083] Perform weighted fusion of task-related features and task-irrelevant features to obtain preliminary fusion features, and perform feature recalibration and channel scale adjustment on the preliminary fusion features to obtain fused spatiotemporal feature representation;

[0084] Hierarchical action recognition is performed on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

[0085] Specifically, the initial graph structure is constructed based on the weighted keyframe feature sequence. In the initial graph structure, each keyframe is represented as a node, and the edge weights between nodes are determined by the spatiotemporal distance and feature similarity between nodes. The spatiotemporal distance can be obtained by calculating the time interval and spatial position change between keyframes, while the feature similarity can be expressed by calculating the cosine similarity between the keyframe feature vectors:

[0086] ;

[0087] in, It is and The feature similarity of key frames, and They are and The feature vector of each key frame. Generate an edge weight matrix based on the spatiotemporal distance and feature similarity. Normalize the edge weight matrix to obtain a standardized adjacency matrix to ensure that the adjacency relationship of each node has a consistent influence in the graph structure. The standardized adjacency matrix can be calculated using the following formula: ;

[0088] in, is the normalized adjacency matrix, is the original edge weight matrix, is the node degree matrix, whose diagonal elements Representation Node The degree of a node is the sum of the weights of the edges connected to it. Multi-layer graph convolution operations are performed on the weighted keyframe feature sequence according to the standardized adjacency matrix to obtain graph convolution features. The graph convolution operation can capture the local features between nodes by applying the convolution kernel on the graph structure. The calculation formula for graph convolution is: ;

[0089] in, It is The feature representation of the layer, It is The weight matrix of the layer, is the activation function. Through multi-layer graph convolution operations, high-level graph convolution features are obtained. The graph convolution features are subjected to residual connection and layer normalization to obtain enhanced graph features. The purpose of residual connection is to retain the original feature information and avoid information loss in deep networks. The formula for residual connection is: Layer normalization is to standardize the features so that the features of each layer have the same distribution, improving the stability and convergence speed of training. According to the current task type, an adaptive feature fuser is constructed to obtain dynamic gating parameters. The adaptive feature fuser dynamically adjusts the fusion method of features according to the task type. The dynamic gating parameters can be calculated through a gating network, which generates a set of weight parameters based on the current input features to control the fusion ratio of different features. The calculation formula of the dynamic gating parameters is:

[0090] ;

[0091] in, is the dynamic gating parameter, and are the weights and biases of the gating network, is the activation function. The feature operation of the enhanced graph features and the dynamic gating parameters is performed to obtain the task-related features. The task-related features are obtained by multiplying the enhanced graph features with the dynamic gating parameters: in, is a task-related feature, Denotes element-by-element product. The task-related features and weighted keyframe feature sequences are decoupled to obtain task-related and task-irrelevant features. The purpose of feature decoupling is to decompose features into task-related and task-irrelevant parts to make better use of feature information. The task-related features and task-irrelevant features are weightedly fused to obtain preliminary fused features. Weighted fusion combines the advantages of task-related and task-irrelevant features by assigning different weights to them:

[0092] ;

[0093] in, is the initial fusion feature, and They are task-related and task-irrelevant features, and is the fusion weight parameter. The preliminary fusion features are recalibrated and the channel scale is adjusted to obtain the fused spatiotemporal feature representation. Feature recalibration adjusts the importance of features so that important features occupy a larger proportion in the fusion features; channel scale adjustment changes the number of channels of the features to make the feature representation more compact and effective. Hierarchical action recognition is performed on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object. Hierarchical action recognition identifies the action characteristics of the monitored object in different time scales and spatial ranges by refining the features layer by layer. Through the hierarchical recognition method, the behavior and dynamics of the monitored object are comprehensively analyzed to generate a detailed analysis report.

[0094] In a specific embodiment, the execution step performs hierarchical action recognition on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object, which may specifically include the following steps:

[0095] Perform convolutional feature operations on the fused spatiotemporal feature representation to obtain a coarse-grained action category probability distribution, and select the corresponding fine-grained classifier based on the coarse-grained action category probability distribution;

[0096] The fine-grained classifier is used to predict the action category of the fused spatiotemporal feature representation to obtain the fine-grained action category prediction result;

[0097] Match the fine-grained action category prediction results according to the predefined action prototype library to obtain standardized action descriptions, and perform meta-learning on the standardized action descriptions to build a target action recognition model;

[0098] Recognize the newly emerged action categories according to the target action recognition model to obtain updated action recognition results, and perform Bayesian analysis on the updated action recognition results to obtain confidence estimates of action predictions;

[0099] Perform class activation mapping on the updated action recognition results according to the confidence estimate of the action prediction to obtain a visualization dataset for action recognition;

[0100] Generate a comprehensive video analysis report of the monitored object based on the updated action recognition results, confidence estimates of action predictions, and visualization datasets of action recognition.

[0101] Specifically, convolutional feature operations are performed on the fused spatiotemporal feature representation, and high-level features are extracted through convolutional neural networks to obtain coarse-grained action category probability distribution. The purpose of convolutional feature operations is to extract important action information in the temporal and spatial dimensions. Assuming the input feature is , the convolution operation can be expressed as:

[0102] ;

[0103] in, It is the feature representation after convolution, Conv represents the convolution operation, is the convolution kernel, is the bias term. Through the convolution operation, the probability distribution of each action category is obtained. According to the coarse-grained action category probability distribution, the corresponding fine-grained classifier is selected. The coarse-grained action category probability distribution provides a preliminary classification result for each large category of action, and then the corresponding fine-grained classifier is selected based on this result for more detailed classification. The selection of fine-grained classifier is based on the maximum probability of each coarse-grained category. The fine-grained classifier is used to predict the action category of the fused spatiotemporal feature representation to obtain the fine-grained action category prediction result. The fine-grained classifier further refines the classification and refines the action to a more specific category. The formula for fine-grained classification is: ;

[0104] in, is the probability distribution of fine-grained action categories, and are the weights and biases of the fine-grained classifier, respectively. The fine-grained action category prediction results are matched according to the predefined action prototype library to obtain standardized action descriptions. The action prototype library contains standardized descriptions of different action categories. By matching the prediction results with the action prototype library, a consistent action description is obtained. Standardized action descriptions help to unify action data from different sources. Meta-learning is performed on the standardized action descriptions to construct a target action recognition model. Meta-learning is a method to improve the performance of a model in a new task by learning shared knowledge between different tasks. The target action recognition model is trained based on standardized action descriptions and can quickly adapt to new action categories. The training process of meta-learning can be expressed as:

[0105] ;

[0106] in, are model parameters, is the task distribution, and L is the loss function. According to the target action recognition model, the newly emerged action categories are identified to obtain updated action recognition results. The target action recognition model can quickly adapt to new action categories and improve the accuracy of recognition. The updated action recognition results are subjected to Bayesian analysis to obtain the confidence estimate of the action prediction. Bayesian analysis estimates the credibility of the action recognition results by calculating the posterior probability. The Bayesian confidence estimation formula is: ;

[0107] in, is the posterior probability, is the likelihood function, is the prior probability, is the marginal likelihood. According to the confidence estimate of the action prediction, the updated action recognition results are subjected to class activation mapping to obtain a visual data set for action recognition. The class activation mapping makes the action recognition results more intuitive and easier to understand by highlighting the important areas in the feature map. The formula for the class activation mapping is: ;

[0108] in, is the class activation map, Yes Category For The weight of the feature map, It is The feature map is at position Based on the updated action recognition results, confidence estimates of action predictions, and visualization datasets of action recognition, a comprehensive video analysis report of the monitored object is generated.

[0109] The video analysis method of the monitoring terminal in the embodiment of the present application is described above. The video analysis system 10 of the monitoring terminal in the embodiment of the present application is described below. Figure 2 In one embodiment of the present application, a video analysis system 10 for monitoring a terminal includes:

[0110] A modeling module 11 is used to perform adaptive frame rate adjustment and background modeling on the real-time video stream collected by the monitoring terminal to obtain an adaptive background model and a target detection result including the monitored object;

[0111] A processing module 12 is used to perform feature processing on the target detection result based on a multi-scale time receptive field network to obtain a multi-scale time feature tensor;

[0112] The analysis module 13 is used to perform coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map, and to perform temporal attention allocation on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence;

[0113] The recognition module 14 is used to perform adaptive feature fusion on the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and perform hierarchical action recognition on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

[0114] Through the collaboration of the above components, adaptive frame rate adjustment and background modeling technology can dynamically adjust the processing strategy according to the scene complexity and terminal computing power, effectively balancing the analysis efficiency and resource consumption. The introduction of the multi-scale temporal receptive field network enables the system to capture short-term and long-term temporal features at the same time, improving the recognition ability of actions of different durations. The combination of coordinate attention and pyramid pooling enhances the expression ability of spatial features while reducing the interference of background noise. Nonlinear kernel function density peak clustering and temporal attention allocation mechanism can effectively extract key frames and assign different importance to different frames, improving the efficiency of feature representation. Graph convolution operation and adaptive feature fusion technology make full use of the complementarity of spatiotemporal information and enhance the discriminative ability of features. The hierarchical action recognition strategy combined with meta-learning and Bayesian analysis not only improves the accuracy of action recognition of the monitoring terminal for the monitored object, but also enhances the model's adaptability to new action categories. This method adopts a series connection of multiple lightweight modules to reduce the computational complexity while ensuring analysis performance, and is suitable for deployment on edge devices.

[0115] The present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the video analysis method of the monitoring terminal in the above-mentioned embodiments.

[0116] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the video analysis method of the monitoring terminal.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0119] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video analysis method for a monitoring terminal, characterized in that: The video analysis method of the monitoring terminal comprises: Adaptively adjust the frame rate and model the background of the real-time video stream collected by the monitoring terminal to obtain an adaptive background model and a target detection result containing the monitored object; specifically, the method includes: evaluating the scene complexity of the real-time video stream collected by the monitoring terminal to obtain a scene complexity index, and dynamically adjusting the frame rate of the real-time video stream according to the scene complexity index and the computing power of the monitoring terminal to obtain an adjusted video stream; performing Gaussian mixture modeling on the adjusted video stream to obtain an initial background model, and performing inter-frame difference processing on the initial background model and the adjusted video stream to obtain a motion region mask; updating the initial background model according to the motion region mask to obtain an adaptive background model; performing target detection on the adjusted video stream through the adaptive background model to obtain an initial detection result, and constructing a region proposal network according to the initial detection result to obtain a candidate region set; performing feature analysis on the candidate region set to obtain a target focus region, and performing cross-frame target tracking processing on the initial detection result according to the target focus region to obtain a target detection result containing the monitored object; The target detection result is subjected to feature processing based on a multi-scale time receptive field network to obtain a multi-scale time feature tensor; specifically comprising: dividing the real-time video stream into time windows based on the multi-scale time receptive field network and the target detection result to obtain a short-term time window sequence and a long-term time window sequence; processing the short-term time window sequence through a 3D convolutional neural network to obtain a short-term time feature, and processing the long-term time window sequence through an expanded 3D convolutional network to obtain a long-term time feature; constructing an adaptive fusion module according to the short-term time feature and the long-term time feature, and calculating a fusion weight coefficient, and weighted fusion of the short-term time feature and the long-term time feature according to the fusion weight coefficient to obtain an initial multi-scale time feature; performing dynamic frame sampling on the initial multi-scale time feature according to the activity level of the scene to obtain a sampled time feature, and performing motion vector analysis on adjacent frames to obtain high activity area positioning information; performing regional weighting on the sampled time feature according to the high activity area positioning information to obtain a regional weighted time feature, and performing feature channel reorganization on the regional weighted time feature to obtain a multi-scale time feature tensor; Performing coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map, and performing temporal attention allocation on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence; Adaptive feature fusion is performed on the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and hierarchical action recognition is performed on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

2. The video analysis method of the monitoring terminal according to claim 1, characterized in that: The method of performing coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map, and performing temporal attention allocation on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence includes: Performing grid segmentation on the multi-scale temporal feature tensor to obtain a feature grid unit set, and performing local attention weight calculation on each unit in the feature grid unit set to obtain a local attention map; Performing global feature extraction on the multi-scale temporal feature tensor to obtain a global feature vector, and performing global and local attention weight analysis based on the local attention map and the global feature vector to obtain a coordinate attention matrix; Performing attention enhancement processing on the multi-scale temporal feature tensor and the coordinate attention matrix to obtain an attention enhancement feature map, and performing a multi-branch parallel pooling operation on the attention enhancement feature map to obtain a multi-scale spatial feature set; Performing feature splicing on the multi-scale spatial feature set to obtain a spliced ​​feature map, and performing a contrastive learning loss function operation on the spliced ​​feature map to obtain a discriminative enhancement feature; Dynamically adjusting the dimension of the discriminative enhancement feature according to the complexity of the current scene to obtain an adaptive feature representation, and performing a residual connection between the adaptive feature representation and the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map; Nonlinear kernel function density peak clustering and time attention allocation are performed on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence.

3. The video analysis method of the monitoring terminal according to claim 2, characterized in that: The step of performing nonlinear kernel function density peak clustering and time attention allocation on the optimized spatiotemporal feature graph to obtain a weighted key frame feature sequence includes: Performing radial basis function kernel calculation on the optimized spatiotemporal feature map to obtain a nonlinear feature space mapping, and performing local density and relative distance calculation on the nonlinear feature space mapping to obtain a density-distance matrix; Identifying density peak points according to the density-distance matrix to obtain an initial cluster center set, and performing adaptive bandwidth selection on the initial cluster center set to obtain optimized cluster centers; Clustering the optimized spatiotemporal feature graph according to the optimized cluster center to obtain a key frame candidate set, and performing scene complexity evaluation on the key frame candidate set to obtain a dynamic threshold; The key frame candidate set is screened according to the dynamic threshold to obtain a key frame set, and the key frame set is encoded in a temporal relationship to obtain a temporal relationship code; A multi-head self-attention mechanism analysis is performed on the temporal relationship encoding to obtain a temporal attention weight, and a weighted analysis is performed on the key frame set and the temporal attention weight to obtain a weighted key frame feature sequence.

4. The video analysis method of the monitoring terminal according to claim 1, characterized in that: The adaptive feature fusion is performed on the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and hierarchical action recognition is performed on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object, including: Constructing an initial graph structure according to the weighted key frame feature sequence, and calculating the spatiotemporal distance and feature similarity between nodes according to the initial graph structure to generate an edge weight matrix; Normalizing the edge weight matrix to obtain a standardized adjacency matrix, and performing a multi-layer graph convolution operation on the weighted key frame feature sequence according to the standardized adjacency matrix to obtain a graph convolution feature; Performing residual connection and layer normalization processing on the graph convolution features to obtain enhanced graph features, and constructing an adaptive feature fuser according to the current task type to obtain dynamic gating parameters; Performing feature operations on the enhanced graph features and the dynamic gating parameters to obtain task-related features, and performing feature decoupling on the task-related features and the weighted key frame feature sequence to obtain task-related and task-independent features; Performing weighted fusion on the task-related features and the task-irrelevant features to obtain preliminary fusion features, and performing feature recalibration and channel scale adjustment on the preliminary fusion features to obtain fusion spatiotemporal feature representation; Hierarchical action recognition is performed on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

5. The video analysis method of the monitoring terminal according to claim 4, characterized in that: The step of performing hierarchical action recognition on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object includes: Performing convolutional feature operations on the fused spatiotemporal feature representation to obtain a coarse-grained action category probability distribution, and selecting a corresponding fine-grained classifier according to the coarse-grained action category probability distribution; Performing action category prediction on the fused spatiotemporal feature representation by the fine-grained classifier to obtain a fine-grained action category prediction result; Matching the fine-grained action category prediction results according to a predefined action prototype library to obtain a standardized action description, and performing meta-learning processing on the standardized action description to construct a target action recognition model; Identifying newly emerged action categories according to the target action recognition model to obtain updated action recognition results, and performing Bayesian analysis on the updated action recognition results to obtain confidence estimates for action predictions; Performing class activation mapping on the updated action recognition results according to the confidence estimation of the action prediction to obtain a visualization data set for action recognition; A comprehensive video analysis report of the monitored object is generated based on the updated action recognition results, the confidence estimate of the action prediction and the visualization data set of the action recognition.

6. A video analysis system for a monitoring terminal, characterized in that: Used to execute the video analysis method of the monitoring terminal according to any one of claims 1 to 5, the video analysis system of the monitoring terminal comprises: A modeling module is used to perform adaptive frame rate adjustment and background modeling on the real-time video stream collected by the monitoring terminal to obtain an adaptive background model and target detection results including the monitored object; A processing module, used for performing feature processing on the target detection result based on a multi-scale time receptive field network to obtain a multi-scale time feature tensor; An analysis module, configured to perform coordinate attention and pyramid pooling processing on the multi-scale temporal feature tensor to obtain an optimized spatiotemporal feature map, and perform temporal attention allocation on the optimized spatiotemporal feature map to obtain a weighted key frame feature sequence; The recognition module is used to perform adaptive feature fusion on the weighted key frame feature sequence to obtain a fused spatiotemporal feature representation, and perform hierarchical action recognition on the fused spatiotemporal feature representation to obtain a comprehensive video analysis report of the monitored object.

7. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes the video analysis method for the monitoring terminal according to any one of claims 1 to 5.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the video analysis method for the monitoring terminal according to any one of claims 1 to 5 is implemented.

Images