Network camera monitoring and identification method and system based on artificial intelligence

Abstract

The application provides a network camera monitoring recognition method and system based on artificial intelligence, first acquires the monitoring data stream containing video frame sequence and corresponding time sequence metadata sequence output by the network camera, then carries out space-time context coding processing on the monitoring data stream, generates a context feature cube containing space position information and time evolution information, then carries out normal behavior mode learning operation based on the context feature cube, generates a reference feature library containing typical scene feature template and feature evolution law description, dynamically matches and compares the context feature cube of the current time period with the reference feature library, calculates the feature matching deviation value and generates the abnormal confidence score, finally generates the monitoring early warning instruction containing the abnormal occurrence time, space coordinate range and confidence level identification according to the abnormal confidence score and corresponding space-time position information, effectively improves the accuracy and early warning effect of the network camera monitoring recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a network camera monitoring and identification method and system based on artificial intelligence. Background Technology

[0002] In numerous fields such as security monitoring, traffic management, and industrial production monitoring, network cameras play a crucial role as important data acquisition devices. Existing network camera monitoring and identification methods mainly focus on simple analysis of video frames, typically extracting only spatial features such as the shape and color of objects, while ignoring temporal information and camera-related temporal metadata, such as acquisition timestamps, camera pose information, and ambient light intensity.

[0003] This single analytical approach has several limitations. Firstly, due to the lack of temporal correlation, it's difficult to accurately identify abnormal behavior patterns that evolve over time; for example, slowly occurring abnormal changes or periodically occurring anomalies are easily missed. Secondly, the failure to fully utilize temporal metadata makes it difficult to effectively standardize and compare monitoring data under different environmental conditions (such as different light intensities and camera poses), significantly reducing the accuracy and reliability of monitoring identification. Therefore, a network camera monitoring identification method that comprehensively considers spatial, temporal, and temporal metadata is needed to improve the effectiveness of monitoring identification. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide an artificial intelligence-based network camera monitoring and identification method, the method comprising:

[0005] Acquire the monitoring data stream output by the network camera. The monitoring data stream includes a continuously acquired video frame sequence and a corresponding time-series metadata sequence. The time-series metadata sequence includes the acquisition timestamp of each video frame, camera pose information, and a description of ambient light intensity.

[0006] The monitoring data stream is subjected to spatiotemporal context encoding processing, which associates and maps the spatial features of the video frame sequence with the temporal features of the temporal metadata sequence to generate a context feature cube containing spatial location information and temporal evolution information.

[0007] Based on the context feature cube, a normal behavior pattern learning operation is performed. The feature patterns that appear stably in multiple time periods are extracted by an unsupervised learning algorithm to generate a benchmark feature library. The benchmark feature library contains typical scene feature templates and descriptions of feature evolution rules.

[0008] The context feature cube of the current time period is dynamically matched and compared with the benchmark feature library to calculate the feature matching deviation value and generate an anomaly confidence score.

[0009] Based on the anomaly confidence score and the corresponding spatiotemporal location information, a monitoring and early warning instruction is generated. The monitoring and early warning instruction includes the time of anomaly occurrence, the spatial coordinate range, and the confidence level identifier.

[0010] In another aspect, embodiments of the present invention also provide an artificial intelligence-based network camera monitoring and recognition system, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0011] Based on the above, this embodiment of the invention acquires a monitoring data stream containing video frame sequences and corresponding temporal metadata sequences, performs spatiotemporal context encoding on the monitoring data stream, associates and maps the spatial features of the video frame sequences with the temporal features of the temporal metadata sequences, and generates a context feature cube containing spatial location information and temporal evolution information. This effectively integrates multi-dimensional information, improves the completeness and accuracy of feature expression, performs normal behavior pattern learning operations based on the context feature cube, extracts stable feature patterns that appear in multiple time periods through unsupervised learning algorithms, generates a benchmark feature library containing typical scene feature templates and descriptions of feature evolution rules, dynamically matches and compares the context feature cube of the current time period with the benchmark feature library, calculates the feature matching deviation value and generates an anomaly confidence score, which can accurately determine abnormal situations in the monitoring scene. Based on the anomaly confidence score and the corresponding spatiotemporal location information, a monitoring early warning instruction containing the time of anomaly occurrence, spatial coordinate range and confidence level identifier is generated, realizing timely detection, accurate location and accurate evaluation of abnormal events, and improving the accuracy and reliability of network camera monitoring and identification. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the execution flow of the network camera monitoring and recognition method based on artificial intelligence provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of exemplary hardware and software components of the network camera monitoring and recognition system based on artificial intelligence provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1This is a flowchart illustrating an artificial intelligence-based network camera monitoring and recognition method according to an embodiment of the present invention. The following is a detailed description of the artificial intelligence-based network camera monitoring and recognition method.

[0015] Step S110: Obtain the monitoring data stream output by the network camera. The monitoring data stream includes a continuously acquired video frame sequence and a corresponding time-series metadata sequence. The time-series metadata sequence includes the acquisition timestamp of each video frame, camera pose information, and a description of ambient light intensity.

[0016] In detail, in a typical security monitoring scenario, network cameras are deployed in a specific area for continuous monitoring. The network cameras capture images of the monitored area at a pre-set frame rate, thereby generating continuous video frames. These video frames are arranged sequentially according to the order in which they were captured, forming a video frame sequence.

[0017] For each video frame, the network camera system can synchronously record its corresponding temporal metadata. The acquisition timestamp, accurate to a specific point in time, is generated based on the camera's internal clock system and determines when the video frame was captured. Camera pose information is acquired through positioning and attitude sensors mounted on the camera, containing the camera's position coordinates in three-dimensional space and its shooting angle. This information can be represented by a set of vectors, where each dimension corresponds to a different direction and angle in space. Ambient light intensity is measured by a light sensor, reflecting the lighting conditions of the monitored area at that moment, possibly expressed as different levels or continuous numerical ranges. These video frame sequences and their corresponding temporal metadata sequences together constitute the monitoring data stream.

[0018] Step S120: Perform spatiotemporal context encoding on the monitoring data stream, associate and map the spatial features of the video frame sequence with the temporal features of the temporal metadata sequence, and generate a context feature cube containing spatial location information and temporal evolution information.

[0019] After acquiring the monitoring data stream, it needs to undergo spatiotemporal context encoding. The purpose of this encoding is to correlate and map the spatial features contained in the video frame sequence with the temporal features reflected in the temporal metadata sequence. Through this processing, a context feature cube can be generated, which contains the spatial location information of the monitored scene and its evolution over time.

[0020] Step S121: Perform spatial feature extraction processing on the video frame sequence, and use a convolutional neural network to extract local region features and global scene features of each video frame to generate a spatial feature vector sequence.

[0021] Using convolutional neural networks (CNNs) to extract spatial features from video frame sequences is a crucial step. A CNN is a deep learning model specifically designed for processing data with a grid structure, making it well-suited for handling two-dimensional image data like video frames. For each video frame in the sequence, the CNN performs feature extraction layer by layer.

[0022] Step S1211: Input the video frame sequence into the pre-trained basic feature extraction network, and extract shallow visual features through convolutional layers. The shallow visual features include edge contours, texture details and color distribution information.

[0023] The pre-trained basic feature extraction network is trained on a large amount of image data and has learned many general image features. When a sequence of video frames is input into this network, the convolutional layers perform convolution operations on the video frames. The convolution operation involves sliding a series of convolutional kernels across the video frames, performing weighted summations on each local region to extract features at different scales and orientations. During this process, the convolutional layers extract shallow visual features from the video frames, such as edge contour features (which can identify object boundaries), texture detail features (which capture the texture information of object surfaces), and color distribution information (which reflects the color distribution in the video frame).

[0024] Step S1212: Perform multi-scale aggregation processing on the shallow visual features through the spatial pyramid pooling module to generate local region features containing different receptive field ranges.

[0025] The spatial pyramid pooling module performs multi-scale aggregation processing on the previously extracted shallow visual features. It divides the shallow visual features into regions of different sizes and performs pooling operations on each region. Pooling reduces the dimensionality of features while preserving important feature information. By dividing the features at different scales, the spatial pyramid pooling module can generate local region features with different receptive field ranges. The receptive field refers to the size of the input image region that a neuron in a convolutional neural network can understand. Different receptive field sizes can capture object features at different scales; for example, a small receptive field can focus on the detailed features of an object, while a large receptive field can capture the overall shape and structure of an object. Thus, the spatial pyramid pooling module obtains a series of local region features with different scales and receptive field ranges.

[0026] Step S1213: Perform global information compression processing on the shallow visual features through a global average pooling layer to generate global scene features that describe the overall scene layout.

[0027] Global average pooling (GAP) layers compress global information based on shallow visual features. It averages the feature values ​​of each channel across the entire shallow visual feature map, resulting in a single numerical value. In this way, GAP compresses global information from shallow visual features, generating a global scene feature that describes the overall scene layout. This global scene feature reflects the overall distribution and structure of all objects and elements in the video frame, ignoring some local details and focusing more on the overall scene features.

[0028] Step S1214: Concatenate and stitch the local region features with the global scene features to generate a spatial feature vector corresponding to each video frame.

[0029] After obtaining the local region features and global scene features, they need to be concatenated and stitched together. Concatenation and stitching involves joining two feature vectors together in a predetermined order to form a longer feature vector. For each video frame, its corresponding local region feature vector and global scene feature vector are concatenated and stitched together to obtain the spatial feature vector corresponding to that video frame. This spatial feature vector integrates the detailed features of the local region and the overall features of the global scene, and can more comprehensively describe the spatial features of the video frame.

[0030] Step S1215: Arrange the spatial feature vectors according to the acquisition order of the video frames to generate a spatial feature vector sequence.

[0031] Finally, the spatial feature vectors corresponding to each video frame are arranged according to the acquisition order of the video frames. Since the video frames are acquired sequentially in chronological order, the resulting sequence of spatial feature vectors reflects the temporal changes in spatial features of the video frame sequence.

[0032] Step S122: Perform time feature encoding processing on the time-series metadata sequence, convert the acquisition timestamp into a time interval sequence, convert the camera pose information into a spatial coordinate offset sequence, convert the ambient light intensity description into a light change gradient sequence, and generate a time feature vector sequence.

[0033] For time-series metadata sequences, time feature encoding is required. By performing specific transformation operations on different types of time-series metadata, they are converted into feature sequences suitable for subsequent processing.

[0034] Step S1221: Convert the collected timestamps into a time interval sequence.

[0035] The capture timestamp records the specific capture time of each video frame. To better reflect the temporal changes, it is converted into a time interval sequence. Specifically, the time difference between two adjacent capture timestamps is calculated, resulting in a series of time interval values. These time interval values ​​constitute the time interval sequence, reflecting the temporal rhythm and intervals of the video frame captures.

[0036] Step S1222: Convert the camera pose information into a spatial coordinate offset sequence.

[0037] Camera pose information includes the camera's position and angle in three-dimensional space. By calculating the difference between the camera pose information of two adjacent video frames, the spatial coordinate offset can be obtained. Arranging these spatial coordinate offsets in the order of the video frames generates a spatial coordinate offset sequence, which reflects the changes in the camera's position and angle during the shooting process.

[0038] Step S1223: Convert the ambient light intensity description into a light change gradient sequence.

[0039] Ambient light intensity descriptions reflect the lighting conditions of the monitored area. The change in ambient light intensity description between two adjacent video frames is calculated to obtain the light change gradient. These gradients are then arranged in the order of the video frames to generate a light change gradient sequence, which reflects the trend of light intensity variation in the monitored area over time.

[0040] Step S1224: Concatenate the time interval sequence, spatial coordinate offset sequence, and illumination change gradient sequence to generate a time feature vector sequence.

[0041] The previously obtained time interval sequence, spatial coordinate offset sequence, and illumination change gradient sequence are concatenated to form a comprehensive time feature vector sequence, which contains various changes in temporal metadata in the time dimension.

[0042] Step S123: Establish the time alignment relationship between the spatial feature vector sequence and the temporal feature vector sequence. Using the acquisition timestamp of each video frame as the alignment reference, the spatial feature vector and temporal feature vector at the corresponding time point are spliced ​​and fused to generate a spatiotemporal fusion feature vector.

[0043] To correlate spatial and temporal features, a temporal alignment relationship needs to be established between the spatial feature vector sequence and the temporal feature vector sequence. Using the acquisition timestamp of each video frame as the alignment benchmark, the spatial and temporal feature vectors are ensured to correspond temporally. For spatial and temporal feature vectors at the same time point, they are concatenated and fused. Concatenation and fusion involves joining two vectors in a predetermined order to form a new vector. The resulting spatiotemporal fused feature vector integrates both spatial and temporal features, providing a more comprehensive description of the spatial and temporal characteristics of the monitored scene.

[0044] Step S124: Arrange the spatiotemporal fusion feature vectors in chronological order to construct a three-dimensional feature matrix, where the first dimension is the time step, the second dimension is the spatial location coordinates, and the third dimension is the feature dimension, generating a context feature cube containing spatial location information and temporal evolution information.

[0045] The obtained spatiotemporal fusion feature vectors are arranged in chronological order to construct a three-dimensional feature matrix. The first dimension of this matrix represents the time step, reflecting the acquisition order and temporal changes of the video frames; the second dimension represents the spatial coordinates, corresponding to different spatial locations within the monitored area; and the third dimension represents the feature dimension, containing the various feature components of the spatiotemporal fusion feature vectors. Through this construction, a context feature cube containing spatial location information and temporal evolution information is generated. This context feature cube can intuitively display the spatial and temporal feature changes of the monitored scene.

[0046] Step S130: Perform normal behavior pattern learning operation based on the context feature cube, extract feature patterns that appear stably in multiple time periods through unsupervised learning algorithm, and generate a benchmark feature library. The benchmark feature library contains feature templates for typical scenes and descriptions of feature evolution rules.

[0047] After obtaining the context feature cube, a normal behavior pattern learning operation needs to be performed based on it. Specifically, an unsupervised learning algorithm is used here, with the aim of extracting stable feature patterns that appear over multiple time periods from the context feature cube, thereby generating a baseline feature library. This baseline feature library will serve as an important reference for subsequent judgments on whether the monitored scene is abnormal.

[0048] Step S131: Perform time window segmentation on the context feature cube and select sub-cubes with continuous time steps as learning units.

[0049] To facilitate the analysis of feature patterns within the context feature cube, it is first divided into time windows. Based on the time dimension, the context feature cube is divided into multiple sub-cubes according to continuous time steps; each sub-cube is a learning unit. The size of the time window needs to be determined based on the actual monitoring scenario and data characteristics. An appropriate time window size ensures that the learning unit contains sufficient feature information to capture stable patterns, but is not so large that patterns are blurred or ignored. For example, in a high-traffic shopping mall monitoring scenario, if the time window is set too small, it may not cover a complete customer behavior cycle; if the time window is set too large, different types of customer behavior may be mixed together, making it difficult to extract stable feature patterns. Each learning unit contains spatial and temporal feature information within a specific time period.

[0050] Step S132: Perform feature clustering analysis on each learning unit and use density clustering algorithm to identify significantly dense feature clusters, where the feature clusters represent stable feature patterns that recur over multiple time periods.

[0051] For each learning unit, density clustering is used for feature clustering analysis. The core idea of ​​density clustering is to cluster based on the density of data points. In the learning units of the context feature cube, data points represent different feature vectors, which contain spatial and temporal feature information. The density clustering algorithm calculates the density around each data point, dividing high-density areas into different feature clusters. Significantly dense feature clusters represent stable feature patterns that recur over multiple time periods. For example, in a shopping mall monitoring scenario, there may be some fixed customer flow paths and dwell areas. The feature vectors corresponding to these areas will cluster together in different time periods, forming significantly dense feature clusters. Density clustering can accurately identify these feature clusters.

[0052] Step S133: Perform feature statistical processing on the feature cluster, calculate the center vector of the feature cluster as a typical scene feature template, and calculate the distribution variance of the feature vectors within the feature cluster as a feature stability index.

[0053] After identifying feature clusters, they need to be statistically processed. For each feature cluster, its central vector is calculated. The central vector is obtained by summing and averaging all the feature vectors within the cluster across all dimensions. This central vector can serve as a template for typical scene features, representing the typical scene features corresponding to that feature cluster. For example, in a shopping mall monitoring scenario, a feature cluster might correspond to customer activity near the checkout counter. Its central vector would represent typical characteristics of customer activity near the checkout counter, such as dwell time and activity range. Simultaneously, the variance of the feature vectors within the feature cluster is calculated. The variance reflects the degree of dispersion of the feature vectors within the cluster. The smaller the variance, the more stable the feature vectors within the cluster. This variance can be used as a feature stability index to measure the stability of the feature pattern. If the variance of a feature cluster is small, it indicates that the feature pattern is very stable over multiple time periods and is more likely to be part of a normal behavioral pattern.

[0054] Step S134: Perform correlation analysis on feature clusters in adjacent time windows, track the evolution trajectory of the same feature cluster in the time dimension, calculate the time change rate and directional offset of feature vectors, and generate a description of feature evolution law.

[0055] To understand the evolution of feature patterns over time, it is necessary to perform correlation analysis on feature clusters in adjacent time windows.

[0056] Step S1341: Obtain the feature cluster set of the previous time window and the feature cluster set of the current time window.

[0057] The corresponding feature cluster sets are extracted from two adjacent time windows. These feature cluster sets contain feature cluster information identified in different time windows, and each feature cluster consists of its center vector and internal feature vectors.

[0058] Step S1342: Perform similarity calculation processing on each feature cluster of the previous time window and the feature cluster of the current time window, and use the cosine similarity algorithm to calculate the similarity of the cluster center vectors.

[0059] For each feature cluster in the previous time window and each feature cluster in the current time window, the cosine similarity algorithm is used to calculate the similarity of their cluster center vectors. Cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them. Specifically, the calculation involves first calculating the dot product of the two cluster center vectors, and then dividing the dot product by the product of the magnitudes of the two vectors. The closer the cosine similarity value is to 1, the more similar the two vectors are; the closer it is to -1, the more opposite the two vectors are; and the closer it is to 0, the less correlation there is between the two vectors. Through this calculation, it can be determined whether two feature clusters might represent the same feature pattern at different times.

[0060] Step S1343: Establish the temporal association relationship of feature clusters based on the similarity calculation results, and mark feature clusters with similarity greater than a preset threshold as the time continuation of the same feature.

[0061] Based on the similarity calculation results, a preset threshold is set. If the similarity between two feature clusters is greater than this preset threshold, they are considered to be continuations of the same feature at different times, and are marked as having a temporal correlation. The preset threshold needs to be adjusted according to the actual situation; an appropriate threshold can accurately establish the temporal correlation between feature clusters. For example, in a shopping mall monitoring scenario, if the similarity between two feature clusters is greater than 0.8, they are considered to be continuations of the same customer behavior pattern at different time periods.

[0062] Step S1344: For feature clusters with temporal continuity, extract the time-series data of the center vector of the feature cluster to generate the feature evolution trajectory.

[0063] For feature clusters with temporal correlation, extract the sequential data of their center vectors along the time dimension. Arrange these center vectors in chronological order to obtain the feature evolution trajectory of the feature cluster. This trajectory reflects the changes in the feature pattern over time. For example, in a shopping mall monitoring scenario, a feature cluster corresponds to the movement path of customers in the mall, and its feature evolution trajectory can show the changes in the direction and speed of customer movement in different time periods.

[0064] Step S1345: Perform first-order difference calculation on the feature evolution trajectory to obtain the feature change amount of adjacent time steps, and generate the feature time change rate by combining the time interval information.

[0065] The first-order difference calculation is performed on the feature evolution trajectory, which involves calculating the difference between the feature vectors of two adjacent time steps. This difference represents the feature change over those adjacent time steps. Combining this with the time interval information, the feature change is divided by the time interval to obtain the feature time change rate. The feature time change rate reflects the speed at which the feature pattern changes over time. For example, in a shopping mall monitoring scenario, if a feature cluster corresponds to a customer's dwell time, the feature time change rate can reflect the speed at which the customer's dwell time changes—whether it is gradually increasing or decreasing.

[0066] Step S1346: Perform direction vector calculation processing on the feature evolution trajectory, extract the main direction component of feature change, and generate feature direction offset.

[0067] The direction vector of the feature evolution trajectory is calculated. By analyzing the changes in the feature vector across different dimensions, the principal directional component of the feature change is extracted. Specifically, two adjacent feature vectors in the feature evolution trajectory are subtracted to obtain a single vector, which is then normalized to obtain the direction vector. The principal directional component of this direction vector is extracted; this principal directional component is the feature direction offset, reflecting the direction of change of the feature pattern over time. For example, in a shopping mall monitoring scenario, if a feature cluster corresponds to the direction of customer movement, the feature direction offset can display changes in the customer's movement direction.

[0068] Step S1347: Fuse the feature time change rate and feature direction offset to generate a feature evolution law description that describes the evolution law of the feature over time.

[0069] The feature time change rate and feature direction offset are fused to form a comprehensive descriptive information. A new vector is obtained by concatenating the feature time change rate and feature direction offset according to a set weight. This vector contains the evolution law of the feature pattern over time, including the rate and direction of change. This feature evolution description will be used as part of the benchmark feature library for subsequent anomaly detection.

[0070] Step S135: Integrate and store the typical scene feature templates, feature stability indicators, and feature evolution rules to generate a benchmark feature library.

[0071] Finally, the typical scenario feature templates, feature stability indicators, and feature evolution law descriptions are integrated and stored. For example, these can be stored in a database, with each feature cluster corresponding to one record containing the typical scenario feature template, feature stability indicators, and feature evolution law description. This information together constitutes the benchmark feature library, which contains the typical features and evolution laws of normal behavior patterns in the monitored scenario.

[0072] Step S140: Dynamically match and compare the context feature cube of the current time period with the benchmark feature library, calculate the feature matching deviation value and generate an anomaly confidence score.

[0073] After generating the baseline feature library, the context feature cube of the current time period needs to be dynamically matched and compared with the baseline feature library to detect any anomalies.

[0074] Step S141: Extract the current feature vector sequence from the context feature cube of the current time period, wherein the current feature vector sequence contains the spatial feature vector and the temporal feature vector of the current time step.

[0075] The current feature vector sequence is extracted from the context feature cube for the current time period. Specifically, based on the time dimension of the context feature cube, feature vectors corresponding to the current time step are extracted. These feature vectors contain spatial and temporal features at the current time point. For example, in a shopping mall monitoring scenario, spatial feature vectors might contain the distribution of people in different areas of the mall, while temporal feature vectors might contain the current trend of pedestrian traffic. These feature vectors are then arranged in chronological order to obtain the current feature vector sequence.

[0076] Step S142: Perform feature block processing on the current feature vector sequence, dividing the feature vectors with continuous time steps into comparison units of the same size as the learning units of the benchmark feature library.

[0077] To facilitate matching and comparison with a benchmark feature library, the current feature vector sequence undergoes feature block processing. Feature vectors with continuous time steps are divided into comparison units of the same size as the learning units in the benchmark feature library. This ensures that the comparison units and learning units are consistent in time and feature dimensions, facilitating effective matching and comparison. For example, in a shopping mall monitoring scenario, if the benchmark feature library's learning units are divided according to a 10-minute time step, then the current feature vector sequence is also divided into blocks of 10-minute time steps, resulting in multiple comparison units.

[0078] Step S143: Perform a benchmark matching operation on each comparison unit, calculate the Euclidean distance between the feature vector within the comparison unit and the corresponding typical scene feature template in the benchmark feature library, and generate a feature matching deviation value.

[0079] For each comparison unit, a benchmark matching operation is performed. The Euclidean distance between the feature vector within the comparison unit and the corresponding typical scene feature template in the benchmark feature library is calculated. Euclidean distance is a commonly used method to measure the distance between two vectors, reflecting their similarity in the feature space. Specifically, the calculation process involves first calculating the sum of squares of the differences between the corresponding dimensions of the two vectors (dimensional alignment and unification are required before calculation), then taking the square root of the sum to obtain the Euclidean distance. The calculated Euclidean distance is the feature matching deviation value; a larger deviation value indicates a greater difference between the current feature vector and the typical scene feature template. For example, in a shopping mall monitoring scenario, if a comparison unit corresponds to the activity of people in a certain area of ​​the mall, calculating the Euclidean distance between the feature vector within that comparison unit and the corresponding typical scene feature template in the benchmark feature library, a larger Euclidean distance indicates a significant difference between the activity of people in that area and the normal pattern.

[0080] Step S144: Perform time-weighted processing on the feature matching deviation values, assign enhancement weights to the deviation values ​​of recent time steps, and generate a weighted deviation value sequence.

[0081] To give greater weight to recent feature changes, the feature matching deviation value is time-weighted.

[0082] Step S1441: Determine the length parameter of the time-weighted window, wherein the length parameter is consistent with the time step of the benchmark feature library learning unit.

[0083] First, determine the length parameter of the time-weighted window. This length parameter should be consistent with the time step size of the benchmark feature library's learning units to ensure that the comparison in the time dimension is reasonable. For example, in a shopping mall monitoring scenario, if the benchmark feature library's learning units are divided into 10-minute time steps, then the length of the time-weighted window should also be set to 10 minutes.

[0084] Step S1442: Generate an exponentially decaying weight coefficient based on the time distance between the time step and the current time point. The exponentially decaying weight coefficient decreases exponentially as the time distance increases.

[0085] An exponentially decaying weight coefficient is generated based on the time distance between the time step and the current time point. Specifically, an exponential function is used to generate the weight coefficients, with the time distance between the time step and the current time point as the independent variable. As the time distance increases, the value of the exponential function decreases rapidly, thus achieving exponential decay of the weights. For example, time steps closer to the current time point have larger weight coefficients, while time steps farther away have smaller weight coefficients.

[0086] Step S1443: Multiply the feature matching deviation value by the exponential decay weight coefficient of the corresponding time step to generate a time-weighted deviation value.

[0087] After obtaining the exponential decay weighting coefficients, each feature matching deviation value is multiplied by the weighting coefficient for its corresponding time step. Thus, more recent feature matching deviation values ​​are assigned a larger weight, while more distant deviation values ​​are assigned a smaller weight. This weighting process more accurately reflects the difference between the current monitoring scenario and the normal mode, as recent changes are often more valuable for reference.

[0088] Step S1444: Arrange the time-weighted deviation values ​​in chronological order to generate a weighted deviation value sequence.

[0089] The time-weighted deviation values ​​are arranged in chronological order to form a weighted deviation value sequence. This weighted deviation value sequence takes into account both feature matching deviation values ​​and time factors, and can more comprehensively show the matching differences between the current monitoring scene and the benchmark feature library over a period of time.

[0090] Step S145: Perform statistical analysis on the weighted deviation value sequence, calculate the mean, maximum value and variance of the deviation values, and generate a comprehensive deviation index.

[0091] Statistical analysis of the weighted deviation value sequence aims to assess the degree of difference between the current monitoring scenario and the normal mode from multiple perspectives. The process involves calculating the mean of the weighted deviation value sequence by summing all deviation values ​​and dividing by the number of deviation values. The mean provides an average deviation level, reflecting the overall difference. The maximum value is also calculated; this is the largest deviation value in the sequence, identifying the point of greatest difference from the normal mode during that period, which helps in detecting potentially serious anomalies. The variance is calculated by squareding the difference between each deviation value and the mean, summing these squares, and dividing by the number of deviation values. Variance measures the dispersion of deviation values; a larger variance indicates greater fluctuations in deviation values, potentially suggesting unstable anomalies. These statistical values ​​are combined to generate a comprehensive deviation index, which more comprehensively reflects the degree of anomaly in the current monitoring scenario.

[0092] Step S146: Input the comprehensive deviation index into a preset confidence mapping function, and generate an anomaly confidence score based on the positive correlation between the deviation value and the probability of anomaly.

[0093] The preset confidence mapping function was determined based on a large amount of experimental data and practical application experience, establishing a mapping relationship between the comprehensive deviation index and the anomaly confidence score. Since the magnitude of the deviation value is positively correlated with the probability of an anomaly, that is, the larger the comprehensive deviation index, the greater the difference between the current monitoring scenario and the normal mode, and the higher the probability of an anomaly occurring.

[0094] Step S1461: Obtain the distribution data of comprehensive deviation index under historical normal scenarios, and calculate the mean deviation and standard deviation under normal scenarios.

[0095] To accurately establish the mapping relationship between the comprehensive deviation index and the anomaly confidence score, it is necessary to obtain the distribution data of the comprehensive deviation index under historical normal scenarios. This data was collected when the monitored scenario was in a normal state and contains a large number of comprehensive deviation index values. Statistical analysis of this data is performed to calculate the mean and standard deviation of the deviation under normal scenarios. The mean deviation is calculated by adding all comprehensive deviation index values ​​and then dividing by the number of data points. The standard deviation is calculated by first squared the difference between each comprehensive deviation index value and the mean deviation, then adding these squared values ​​together and dividing by the number of data points, finally taking the square root. The mean deviation reflects the average level of the comprehensive deviation index under normal conditions, while the standard deviation reflects the dispersion of these deviation values ​​around the mean.

[0096] Step S1462: Based on the mean deviation, divide the difference between the current comprehensive deviation index and the mean deviation by the standard deviation of the deviation to generate a standardized deviation value.

[0097] Using the mean deviation under normal conditions as a benchmark, the difference between the current comprehensive deviation index and the mean deviation is divided by the standard deviation of the deviation to obtain the standardized deviation value. This standardization process can transform comprehensive deviation indices of different magnitudes into comparable values, eliminating the impact of differences in the magnitude of comprehensive deviation indices across different monitoring scenarios or time periods. The standardized deviation value can more accurately reflect the degree of deviation of the current monitoring scenario from the normal mode.

[0098] Step S1463: Convert the standardized deviation value into a confidence score in the interval [0, 1] using a nonlinear mapping method, where the larger the standardized deviation value, the higher the outlier confidence score.

[0099] A nonlinear mapping method is used to convert standardized deviation values ​​into anomaly confidence scores in the [0, 1] interval. The nonlinear mapping function can be designed according to the actual situation, and usually a monotonically increasing function is used, so that the larger the standardized deviation value, the higher the corresponding anomaly confidence score. For example, an S-shaped curve function can be used as the nonlinear mapping function. This nonlinear mapping function increases slowly when the standardized deviation value is small, and the function value rises rapidly as the standardized deviation value increases, eventually approaching 1. Through the above nonlinear mapping, the standardized deviation values ​​can be more reasonably converted into anomaly confidence scores, so that the scores can accurately reflect the probability of anomalies.

[0100] Step S150: Generate a monitoring and early warning instruction based on the anomaly confidence score and the corresponding spatiotemporal location information. The monitoring and early warning instruction includes the time of anomaly occurrence, the spatial coordinate range, and the confidence level identifier.

[0101] After obtaining the anomaly confidence score and the corresponding spatiotemporal location information, it is necessary to generate a monitoring and early warning instruction based on this information. The monitoring and early warning instruction should include the time of the anomaly, the spatial coordinate range, and the confidence level indicator, so as to provide monitoring personnel with a comprehensive and accurate description of the anomaly and enable them to take appropriate measures in a timely manner.

[0102] Step S151: Extract the timestamp information corresponding to the anomaly confidence score from the context feature cube to determine the start and end times of the anomaly.

[0103] The context feature cube records the feature information corresponding to each time step and the associated anomaly confidence score. The goal is to extract the timestamp information corresponding to the anomaly confidence score. First, a threshold for the anomaly confidence score is set. When the anomaly confidence score for a given time step exceeds this threshold, it is considered a time point where an anomaly may occur. From these time points exceeding the threshold, the earliest time point is identified as the start time of the anomaly, and the latest time point as the end time. For example, in a shopping mall monitoring scenario, if the anomaly confidence score threshold is set to 0.6, and the anomaly confidence score exceeds 0.6 from the 10th time step until the 20th time step, then the timestamp corresponding to the 10th time step is the start time of the anomaly, and the timestamp corresponding to the 20th time step is the end time.

[0104] Step S152: Based on the spatial location information of the context feature cube, locate the spatial region where the feature matching deviation value exceeds a preset threshold, and generate the spatial coordinate range where the anomaly occurred.

[0105] The second dimension of the context feature cube represents spatial location coordinates. For each spatial location coordinate, the corresponding result has already been obtained when calculating the feature matching deviation value. A preset feature matching deviation value threshold is set, and the feature matching deviation value corresponding to each spatial location in the context feature cube is iterated. When the feature matching deviation value of a spatial location exceeds the threshold, it is marked as a potentially abnormal spatial location. All marked spatial locations are integrated to determine their boundary range, thereby generating the spatial coordinate range where the anomaly occurs. Taking shopping mall monitoring as an example, the mall is divided into multiple areas, each with corresponding spatial coordinates. When the feature matching deviation value of a certain area exceeds the preset threshold, that area is included in the abnormal spatial range. Finally, all abnormal areas are combined to obtain a spatial coordinate range containing multiple coordinate points, such as represented by a rectangular or polygonal area.

[0106] Step S153: Divide the abnormal confidence score into multiple continuous intervals, each interval corresponding to a confidence level identifier, and the confidence level identifier increases as the confidence score increases.

[0107] Based on extensive experimental data and practical application experience, the anomaly confidence score within the 0-1 range is divided into multiple consecutive intervals. For example, it can be divided into three intervals: [0, 0.3), [0.3, 0.7), and [0.7, 1]. Each interval is assigned a corresponding confidence level label, such as "mild anomaly" corresponding to the [0, 0.3) interval, "moderate anomaly" corresponding to the [0.3, 0.7) interval, and "severe anomaly" corresponding to the [0.7, 1] interval. Thus, the confidence level label increases with the anomaly confidence score, intuitively reflecting the severity of the anomaly.

[0108] Step S154: Determine the corresponding confidence level identifier based on the interval to which the abnormal confidence score belongs.

[0109] The calculated anomaly confidence score is compared with the defined intervals to determine the interval to which the score belongs. Once the interval is determined, the severity of the anomaly can be determined based on the corresponding confidence level label. For example, if the anomaly confidence score is 0.8, it belongs to the interval [0.7, 1], and the corresponding confidence level label is "severe anomaly".

[0110] Step S155: Integrate the start and end times of the anomaly, the spatial coordinate range, and the confidence level identifier to generate a monitoring and early warning instruction containing spatiotemporal location information and severity information.

[0111] The start and end times, spatial coordinate range, and confidence level indicators of the identified anomalies are integrated. A structured approach can be used to generate monitoring and early warning instructions, such as a text-based statement: "During the period [start time] - [end time], an anomaly with a [confidence level indicator] occurred in the [spatial coordinate range] area. Please handle it promptly." In a shopping mall monitoring scenario, the monitoring and early warning instruction might be: "During 10:00-10:30, a serious anomaly occurred in the area on the east side of the third floor of the mall (coordinate range: [x1, y1, z1] - [x2, y2, z2]). Please handle it promptly." Such monitoring and early warning instructions provide monitoring personnel with clear spatiotemporal location information and anomaly severity information, facilitating rapid decision-making and appropriate measures.

[0112] Step S210: Obtain historical monitoring calibration dataset, which includes monitoring data streams for known normal scenarios and monitoring data streams for known abnormal scenarios.

[0113] To accurately calibrate the surveillance recognition model, historical surveillance calibration datasets are needed. In practice, data can be collected from multiple data sources. For surveillance data streams of known normal scenarios, collection can be conducted during periods when the surveillance scenario is stable and without anomalies. For example, in a shopping mall surveillance scenario, normal business hours on a weekday can be selected. At this time, the flow of people and the operation of equipment in the mall are in a normal state, and network cameras continuously record video frame sequences and corresponding time-series metadata sequences. These data constitute the surveillance data stream of a normal scenario. For surveillance data streams of known abnormal scenarios, various possible abnormal situations can be simulated to collect data. For example, simulated abnormal events such as fires and thefts in the mall, while recording the surveillance data streams when these abnormal events occur. When collecting data, it is important to ensure the integrity and accuracy of the data. The collected data should be initially screened and cleaned to remove invalid or erroneous data to ensure the quality of the historical surveillance calibration dataset.

[0114] Step S220: Perform context encoding processing on the historical monitoring calibration dataset to generate normal scene context feature cubes and abnormal scene context feature cubes.

[0115] Context encoding is performed on the historical monitoring calibration dataset, a process similar to that of real-time monitoring data streams. For monitoring data streams of normal scenes, spatial feature extraction is first performed on the video frame sequence. A convolutional neural network is used, inputting the video frames into a pre-trained basic feature extraction network. The pre-trained network performs multiple convolution operations on the video frames through convolutional layers, extracting shallow visual features containing edge contours, texture details, and color distribution information. Next, a spatial pyramid pooling module aggregates the shallow visual features at multiple scales, generating local region features with different receptive fields. Then, a global average pooling layer is used to compress the shallow visual features globally, obtaining global scene features describing the overall scene layout. The local region features and global scene features are concatenated and concatenated to generate spatial feature vectors corresponding to each video frame, and arranged into a spatial feature vector sequence according to the acquisition order.

[0116] Simultaneously, the temporal metadata sequence undergoes temporal feature encoding. Acquisition timestamps are converted into time interval sequences, and the difference between adjacent timestamps is calculated. Camera pose information is converted into a spatial coordinate offset sequence, and the pose information difference between adjacent frames is calculated. Ambient light intensity descriptions are converted into light change gradient sequences, and the change in light intensity between adjacent frames is calculated. These three sequences are concatenated into a temporal feature vector sequence.

[0117] Then, using the capture timestamp of each video frame as the alignment reference, the spatial feature vector and temporal feature vector at the corresponding time point are concatenated and fused to generate a spatiotemporal fusion feature vector. Finally, the spatiotemporal fusion feature vector is arranged in chronological order to construct a three-dimensional feature matrix and generate a normal scene context feature cube.

[0118] The same processing flow is used for monitoring data streams of abnormal scenarios to ultimately generate an abnormal scenario context feature cube.

[0119] Step S230: Perform normal behavior pattern learning operation using the normal scene context feature cube to generate an initial benchmark feature library.

[0120] Normal behavior pattern learning is performed using a normal scene context feature cube. First, the normal scene context feature cube is divided into time windows, and the appropriate time window length is determined based on the characteristics and requirements of the monitoring scene. For example, in shopping mall monitoring, the time window can be set to 10 minutes, and sub-cubes with consecutive 10-minute time steps can be selected as learning units.

[0121] For each learning unit, feature clustering analysis is performed using a density-based clustering algorithm. This algorithm first defines a radius parameter and a minimum number of points parameter. For each feature point in the learning unit, the number of points within the radius is calculated. If the number of points exceeds the minimum number of points parameter, that point is considered a core point, and its surrounding points are clustered into the same cluster. By continuously expanding the neighborhood of the core points, significantly dense feature clusters are identified. These feature clusters represent stable feature patterns that recur over multiple time periods.

[0122] Feature statistical processing is performed on the feature clusters to calculate the center vector of the feature clusters. The center vector is obtained by summing each dimension of all feature vectors within the feature cluster and then dividing by the number of feature vectors. This center vector is used as a feature template for a typical scene. Simultaneously, the variance of the feature vector distribution within the feature cluster is calculated to measure the dispersion of the feature vectors relative to the center vector, serving as a feature stability index.

[0123] A correlation analysis is performed on feature clusters in adjacent time windows. The feature cluster sets of the previous and current time windows are obtained. For each feature cluster from the previous and current time windows, a similarity calculation is performed using the cosine similarity algorithm. The dot product of the center vectors of the two clusters is calculated, and then divided by the product of the magnitudes of the two vectors to obtain the similarity value. Based on the similarity calculation results, a temporal correlation relationship between feature clusters is established. A similarity threshold is set; when the similarity exceeds this threshold, the two feature clusters are marked as a temporal continuation of the same feature.

[0124] For feature clusters with temporal continuity, the sequential data of their center vectors in the time dimension are extracted to generate feature evolution trajectories. First-order difference calculations are performed on the feature evolution trajectories to calculate the feature changes at adjacent time steps, and then combined with time interval information to obtain the feature time change rate. Direction vector calculations are performed on the feature evolution trajectories; by analyzing the changing directions of adjacent feature vectors, the principal directional components of feature changes are extracted to generate feature direction offsets. The feature time change rate and feature direction offsets are fused to generate a description of the feature evolution pattern.

[0125] Finally, the typical scenario feature templates, feature stability indicators, and feature evolution rules are integrated and stored to generate an initial benchmark feature library.

[0126] Step S240: Match and compare the abnormal scene context feature cube with the initial benchmark feature library, and calculate the feature matching deviation value distribution under abnormal scene.

[0127] The abnormal scene context feature cube is matched and compared with the initial benchmark feature library. The current feature vector sequence is extracted from the abnormal scene context feature cube; this sequence contains spatial and temporal feature vectors for the current time step. Feature block processing is then performed on the current feature vector sequence, dividing the feature vectors for consecutive time steps into comparison units of the same size as the learning units in the benchmark feature library.

[0128] For each comparison unit, a benchmark matching operation is performed, calculating the Euclidean distance between the feature vector within the comparison unit and the corresponding typical scene feature template in the benchmark feature library. Specifically, the feature vector is subtracted from the corresponding dimension of the typical scene feature template, the sum of squares is taken, and then the square root is obtained to yield the feature matching deviation value. Statistical analysis is performed on the feature matching deviation values ​​of all comparison units, and histograms are plotted or probability density functions are calculated to obtain the distribution of feature matching deviation values ​​under abnormal scenarios.

[0129] Step S250: Based on the difference between the feature matching deviation value distribution in the abnormal scenario and the deviation value distribution in the normal scenario, adjust the parameters of the confidence mapping function so that the abnormal confidence score of the abnormal scenario is higher than that of the normal scenario.

[0130] Step S251: Analyze the set of comprehensive deviation indicators corresponding to the normal scenario to determine the distribution range of deviation values ​​for the normal scenario.

[0131] We analyze the comprehensive deviation index set corresponding to normal scenarios. The comprehensive deviation index is obtained after time-weighting and statistical analysis of the feature matching deviation values ​​in normal scenarios. First, we review the calculation process of feature matching deviation values ​​in normal scenarios, and perform time-weighting on the deviation values, assigning reinforcement weights to the deviation values ​​of recent time steps. We determine the length parameter of the time-weighting window to be consistent with the time step of the benchmark feature library learning unit. Based on the time distance between the time step and the current time point, we generate exponential decay weight coefficients. We multiply the feature matching deviation values ​​by the corresponding weight coefficients to obtain the time-weighted deviation values, and arrange them into a weighted deviation value sequence in chronological order.

[0132] Perform statistical analysis on the weighted deviation value sequence to calculate the mean, maximum, and variance of the deviation values, generating a comprehensive deviation index. Collect a large number of comprehensive deviation indices under normal scenarios, plot histograms or fit probability distribution curves to determine the distribution range of deviation values ​​under normal scenarios, for example, by calculating the standard deviation of the mean plus or minus a set multiple to determine the distribution interval.

[0133] Step S252: Analyze the distribution of feature matching deviation values ​​under the abnormal scenario to determine the range of deviation value distribution under the abnormal scenario.

[0134] The distribution of feature matching deviation values ​​under abnormal scenarios is analyzed. Following the method used to calculate feature matching deviation values ​​for abnormal scenarios, a large number of feature matching deviation values ​​are obtained. These deviation values ​​are then subjected to time-weighted processing and statistical analysis to generate a comprehensive deviation index for abnormal scenarios. A histogram or fitted probability distribution curve of the comprehensive deviation index for abnormal scenarios is plotted to determine the distribution range of deviation values ​​for abnormal scenarios. Similar to the analysis process for normal scenarios, the distribution interval is determined by calculating statistical measures such as the mean and standard deviation.

[0135] Step S253: Adjust the parameters of the confidence mapping function according to the degree of overlap between the deviation value distribution range of the normal scenario and the deviation value distribution range of the abnormal scenario.

[0136] Compare the overlap between the deviation value distribution ranges of normal and abnormal scenarios. If the overlap is significant, it indicates that the current confidence mapping function cannot effectively distinguish between normal and abnormal scenarios, and the function parameters need adjustment. The confidence mapping function is typically a non-linear function, and its parameters determine the mapping relationship between the comprehensive deviation index and the abnormal confidence score. The parameters can be adjusted iteratively. After each adjustment, the abnormal confidence scores for both normal and abnormal scenarios are recalculated, and the change in overlap is observed. For example, increasing the slope of the function in regions with large deviation values ​​will result in a higher abnormal confidence score corresponding to the comprehensive deviation index for abnormal scenarios.

[0137] Step S254: Through an iterative verification process, gradually optimize the parameters until the abnormal confidence scores of normal scenarios are concentrated in the first interval and the abnormal confidence scores of abnormal scenarios are concentrated in the second interval. The highest score value in the first interval is less than the lowest score value in the second interval, and the difference between the highest score value and the lowest score value is greater than a set difference.

[0138] An iterative verification process is conducted to continuously optimize the parameters of the confidence mapping function. After each parameter adjustment, verification is performed using a historical monitoring calibration dataset. The comprehensive deviation indexes for normal and abnormal scenarios are input into the adjusted confidence mapping function to obtain the corresponding abnormal confidence scores. The distribution of abnormal confidence scores for normal and abnormal scenarios is statistically analyzed to observe whether the abnormal confidence scores for normal scenarios are concentrated in the first interval, and the abnormal confidence scores for abnormal scenarios are concentrated in the second interval, with sufficient intervals between the two intervals. The difference is set according to the actual application requirements, for example, it can be set to 0.2. If the conditions are not met, the parameters are adjusted again, and the verification process is repeated until satisfactory results are achieved.

[0139] Step S255: Output the adjusted confidence mapping function parameters.

[0140] After multiple iterations and verifications, when the anomaly confidence scores for normal scenarios are concentrated in the first interval, and the anomaly confidence scores for abnormal scenarios are concentrated in the second interval, and the difference between the highest and lowest scores is greater than a set difference, the adjusted confidence mapping function parameters are output. These parameters will be used for subsequent real-time monitoring data stream processing, enabling the monitoring and identification model to more accurately identify abnormal situations.

[0141] Step S260: Combine the adjusted confidence mapping function with the initial benchmark feature library to generate a calibrated monitoring and identification model.

[0142] The adjusted confidence mapping function is combined with the initial benchmark feature library to form the calibrated monitoring and identification model. In subsequent real-time monitoring, after acquiring the real-time monitoring data stream, a context feature cube is generated following the previous process. This cube is then matched and compared with the initial benchmark feature library to calculate a comprehensive deviation index. This comprehensive deviation index is then input into the adjusted confidence mapping function to obtain a more accurate anomaly confidence score. The calibrated monitoring and identification model integrates feature information from normal scenarios with the optimized confidence mapping relationship, enabling it to more effectively identify anomalies.

[0143] Step S270: Process the real-time monitoring data stream using the calibrated monitoring and identification model to generate an anomaly confidence score.

[0144] The calibrated monitoring and recognition model is used to process the real-time monitoring data stream. First, the real-time monitoring data stream, including video frame sequences and temporal metadata sequences, is acquired. Spatiotemporal context encoding is performed on the real-time monitoring data stream to generate a real-time context feature cube. Then, the real-time context feature cube is dynamically matched and compared with an initial benchmark feature library. The current feature vector sequence is extracted from the real-time context feature cube, and feature block processing is performed, dividing it into comparison units of the same size as the learning units in the benchmark feature library. The Euclidean distance between the feature vectors within the comparison unit and the corresponding typical scene feature templates in the benchmark feature library is calculated to obtain the feature matching deviation value. The feature matching deviation value is then time-weighted to generate a weighted deviation value sequence, and statistical analysis is performed to obtain a comprehensive deviation index.

[0145] Finally, the comprehensive deviation index is input into the adjusted confidence mapping function, and an anomaly confidence score is generated based on the mapping relationship of the function. This anomaly confidence score can more accurately reflect the possibility of anomalies occurring in real-time monitoring scenarios.

[0146] Throughout the data processing, various privacy protection and leak prevention technologies were employed when collecting privacy-sensitive data. The collection of monitoring data streams was only carried out under conditions ensuring legality, compliance, and authorized permission. For the storage of monitoring data streams, encryption technology was used, storing the data in encrypted form on the server, accessible only to authorized users. During data transmission, secure transmission protocols, such as SSL / TLS, were used to encrypt the data and prevent its theft or tampering during transmission. Furthermore, strict identity authentication and access control were implemented for users accessing the data, ensuring that only users with the appropriate authorization could access and process privacy-sensitive data.

[0147] In the above embodiments, the convolutional neural network (CNN) serves as the key model for spatial feature extraction, comprising multiple convolutional layers, pooling layers, and fully connected layers. Convolutional layers extract features from video frames, pooling layers reduce the dimensionality of features, and fully connected layers integrate and classify the features. During training, a large amount of image data is used, and the model parameters are continuously adjusted through backpropagation to ensure the model accurately extracts local and global scene features from video frames. Density clustering, an unsupervised learning algorithm, calculates the density between data points, dividing high-density regions into different clusters. During training, no data labeling is required; the density clustering algorithm automatically identifies feature clusters within the data.

[0148] Figure 2The illustration shows exemplary hardware and software components of an AI-based network camera monitoring and recognition system 100 that can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used in the AI-based network camera monitoring and recognition system 100 and to perform the functions described in this application.

[0149] The AI-based network camera monitoring and recognition system 100 can be a general-purpose server or a special-purpose server; both can be used to implement the AI-based network camera monitoring and recognition method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0150] For example, the AI-based network camera surveillance and recognition system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the AI-based network camera surveillance and recognition system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The AI-based network camera surveillance and recognition system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0151] For ease of explanation, only one processor is described in the AI-based network camera monitoring and recognition system 100. However, it should be noted that the AI-based network camera monitoring and recognition system 100 of this application may also include multiple processors. Therefore, the steps performed by one processor as described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the AI-based network camera monitoring and recognition system 100 performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0152] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned network camera monitoring and recognition method based on artificial intelligence is implemented.

[0153] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A network camera monitoring and recognition method based on artificial intelligence, characterized in that, The method includes: Acquire the monitoring data stream output by the network camera. The monitoring data stream includes a continuously acquired video frame sequence and a corresponding time-series metadata sequence. The time-series metadata sequence includes the acquisition timestamp of each video frame, camera pose information, and a description of ambient light intensity. The monitoring data stream is subjected to spatiotemporal context encoding processing, which associates and maps the spatial features of the video frame sequence with the temporal features of the temporal metadata sequence to generate a context feature cube containing spatial location information and temporal evolution information. Based on the context feature cube, a normal behavior pattern learning operation is performed. The feature patterns that appear stably in multiple time periods are extracted by an unsupervised learning algorithm to generate a benchmark feature library. The benchmark feature library contains typical scene feature templates and descriptions of feature evolution rules. The context feature cube of the current time period is dynamically matched and compared with the benchmark feature library to calculate the feature matching deviation value and generate an anomaly confidence score. Based on the anomaly confidence score and the corresponding spatiotemporal location information, a monitoring and early warning instruction is generated. The monitoring and early warning instruction includes the time of anomaly occurrence, the spatial coordinate range, and the confidence level identifier.

2. The network camera monitoring and identification method based on artificial intelligence according to claim 1, characterized in that, The spatiotemporal context encoding process performed on the monitoring data stream associates and maps the spatial features of the video frame sequence with the temporal features of the temporal metadata sequence to generate a context feature cube containing spatial location information and temporal evolution information, including: Spatial feature extraction processing is performed on the video frame sequence. A convolutional neural network is used to extract local region features and global scene features of each video frame to generate a spatial feature vector sequence. The temporal metadata sequence is processed by temporal feature encoding, which converts the acquisition timestamp into a time interval sequence, the camera pose information into a spatial coordinate offset sequence, and the ambient light intensity description into a light change gradient sequence, thereby generating a temporal feature vector sequence. Establish the time alignment relationship between the spatial feature vector sequence and the temporal feature vector sequence. Using the acquisition timestamp of each video frame as the alignment reference, the spatial feature vector and temporal feature vector at the corresponding time point are spliced ​​and fused to generate a spatiotemporal fusion feature vector. The spatiotemporal fusion feature vectors are arranged in chronological order to construct a three-dimensional feature matrix, where the first dimension is the time step, the second dimension is the spatial location coordinates, and the third dimension is the feature dimension, generating a context feature cube containing spatial location information and temporal evolution information.

3. The network camera monitoring and identification method based on artificial intelligence according to claim 2, characterized in that, The step of performing spatial feature extraction processing on the video frame sequence, using a convolutional neural network to extract local region features and global scene features of each video frame to generate a spatial feature vector sequence, includes: The video frame sequence is input into a pre-trained basic feature extraction network, and shallow visual features are extracted through convolutional layers. The shallow visual features include edge contours, texture details, and color distribution information. The shallow visual features are aggregated at multiple scales using a spatial pyramid pooling module to generate local region features containing different receptive field ranges. The shallow visual features are compressed globally using a global average pooling layer to generate global scene features that describe the overall scene layout. The local region features are concatenated and stitched with the global scene features to generate a spatial feature vector corresponding to each video frame. The spatial feature vectors are arranged according to the acquisition order of the video frames to generate a spatial feature vector sequence.

4. The network camera monitoring and identification method based on artificial intelligence according to claim 1, characterized in that, The normal behavior pattern learning operation is performed based on the context feature cube. An unsupervised learning algorithm is used to extract stable feature patterns that occur across multiple time periods, generating a benchmark feature library. This benchmark feature library contains typical scene feature templates and descriptions of feature evolution patterns, including: The context feature cube is divided into time windows, and sub-cubes with continuous time steps are selected as learning units. Each learning unit is subjected to feature clustering analysis, and density clustering algorithm is used to identify significantly dense feature clusters, which represent stable feature patterns that recur over multiple time periods. The feature clusters are subjected to feature statistical processing. The center vector of the feature cluster is calculated as a typical scene feature template, and the distribution variance of the feature vectors within the feature cluster is calculated as a feature stability index. The feature clusters in adjacent time windows are correlated and processed to track the evolution trajectory of the same feature cluster in the time dimension, calculate the time change rate and directional offset of the feature vector, and generate a description of the feature evolution law. The typical scenario feature templates, feature stability indicators, and feature evolution rules are integrated and stored to generate a benchmark feature library.

5. The network camera monitoring and identification method based on artificial intelligence according to claim 4, characterized in that, The process of performing correlation analysis on feature clusters in adjacent time windows, tracking the evolution trajectory of the same feature cluster in the time dimension, calculating the time change rate and directional offset of the feature vector, and generating a description of the feature evolution law includes: Obtain the feature cluster set of the previous time window and the feature cluster set of the current time window; For each feature cluster in the previous time window and the feature cluster in the current time window, a similarity calculation is performed, and the cosine similarity algorithm is used to calculate the similarity of the cluster center vectors. Establish the temporal association relationship of feature clusters based on the similarity calculation results, and mark feature clusters with similarity greater than a preset threshold as the temporal continuation of the same feature; For feature clusters with temporal continuity, extract the time-series data of the center vector of the feature cluster to generate the feature evolution trajectory; The first-order difference calculation is performed on the feature evolution trajectory to obtain the feature change amount of adjacent time steps, and the feature time change rate is generated by combining the time interval information. The feature evolution trajectory is processed by direction vector calculation to extract the main direction component of feature change and generate feature direction offset. By integrating the feature's time change rate and feature direction offset, a feature evolution law description is generated that describes the feature's evolution over time.

6. The network camera monitoring and identification method based on artificial intelligence according to claim 1, characterized in that, The step of dynamically matching and comparing the context feature cube of the current time period with the benchmark feature library, calculating the feature matching deviation value, and generating an anomaly confidence score includes: Extract the current feature vector sequence from the context feature cube of the current time period, the current feature vector sequence containing the spatial feature vector and the temporal feature vector of the current time step; The current feature vector sequence is processed by feature block segmentation, dividing the feature vectors of continuous time steps into comparison units of the same size as the learning units of the benchmark feature library; Perform a benchmark matching operation on each comparison unit, calculate the Euclidean distance between the feature vector within the comparison unit and the corresponding typical scene feature template in the benchmark feature library, and generate a feature matching deviation value; The feature matching deviation values ​​are time-weighted, and reinforcement weights are assigned to the deviation values ​​of recent time steps to generate a weighted deviation value sequence. The weighted deviation value sequence is statistically analyzed to calculate the mean, maximum value and variance of the deviation values, and a comprehensive deviation index is generated. The comprehensive deviation index is input into a preset confidence mapping function, and an anomaly confidence score is generated based on the positive correlation between the deviation value and the probability of anomaly.

7. The network camera monitoring and identification method based on artificial intelligence according to claim 6, characterized in that, The step of performing time-weighted processing on the feature matching deviation values, assigning enhancement weights to the deviation values ​​of recent time steps, and generating a weighted deviation value sequence includes: Determine the length parameter of the time-weighted window, wherein the length parameter is consistent with the time step of the benchmark feature library learning unit; Based on the time distance between the time step and the current time point, an exponentially decaying weight coefficient is generated, which decreases exponentially with the increase of time distance. The feature matching deviation value is multiplied by the exponential decay weight coefficient of the corresponding time step to generate a time-weighted deviation value; Arrange the time-weighted deviation values ​​in chronological order to generate a weighted deviation value sequence; The step of inputting the comprehensive deviation index into a preset confidence mapping function, and generating an anomaly confidence score based on the positive correlation between the deviation value and the probability of anomaly, includes: Obtain the distribution data of comprehensive deviation indicators under historical normal scenarios, and statistically analyze the mean deviation and standard deviation under normal scenarios; Based on the mean deviation, the difference between the current comprehensive deviation index and the mean deviation is divided by the standard deviation of the deviation to generate the standardized deviation value. The standardized deviation value is converted into a confidence score in the interval [0, 1] by a non-linear mapping method, where the larger the standardized deviation value, the higher the outlier confidence score.

8. The network camera monitoring and identification method based on artificial intelligence according to claim 6, characterized in that, The method includes: Obtain a historical monitoring calibration dataset, which includes monitoring data streams for known normal scenarios and monitoring data streams for known abnormal scenarios; The historical monitoring calibration dataset is subjected to context encoding to generate normal scene context feature cubes and abnormal scene context feature cubes; The normal scene context feature cube is used to perform normal behavior pattern learning operations to generate an initial benchmark feature library. The abnormal scene context feature cube is matched and compared with the initial benchmark feature library to calculate the feature matching deviation value distribution under the abnormal scene; Based on the difference between the feature matching deviation value distribution in the abnormal scenario and the deviation value distribution in the normal scenario, the parameters of the confidence mapping function are adjusted so that the abnormal confidence score of the abnormal scenario is higher than that of the normal scenario. The adjusted confidence mapping function is combined with the initial benchmark feature library to generate a calibrated monitoring and identification model; The calibrated monitoring and identification model is used to process the real-time monitoring data stream to generate an anomaly confidence score. The step of adjusting the parameters of the confidence mapping function based on the difference between the feature matching deviation value distribution in the abnormal scenario and the deviation value distribution in the normal scenario, so that the abnormal confidence score in the abnormal scenario is higher than the abnormal confidence score in the normal scenario, includes: Analyze the set of comprehensive deviation indicators corresponding to the normal scenario to determine the distribution range of deviation values ​​for the normal scenario; Analyze the distribution of feature matching deviation values ​​under the abnormal scenarios to determine the range of deviation value distribution under the abnormal scenarios; The parameters of the confidence mapping function are adjusted based on the degree of overlap between the deviation value distribution range of the normal scenario and the deviation value distribution range of the abnormal scenario. Through an iterative verification process, parameters are gradually optimized until the abnormal confidence scores of normal scenarios are concentrated in the first interval and the abnormal confidence scores of abnormal scenarios are concentrated in the second interval. The highest score value in the first interval is less than the lowest score value in the second interval, and the difference between the highest score value and the lowest score value is greater than a set difference. Output the adjusted confidence mapping function parameters.

9. The network camera monitoring and identification method based on artificial intelligence according to claim 1, characterized in that, The monitoring and early warning instruction is generated based on the anomaly confidence score and the corresponding spatiotemporal location information. The monitoring and early warning instruction includes the anomaly occurrence time, spatial coordinate range, and confidence level identifier, including: Extract the timestamp information corresponding to the anomaly confidence score from the context feature cube to determine the start and end times of the anomaly. Based on the spatial location information of the context feature cube, locate the spatial region where the feature matching deviation value exceeds a preset threshold, and generate the spatial coordinate range of the anomaly occurrence. The abnormal confidence score is divided into multiple continuous intervals, each interval corresponding to a confidence level indicator, and the confidence level indicator increases as the confidence score increases; Based on the interval to which the abnormal confidence score belongs, determine the corresponding confidence level identifier; By integrating the start and end times of the anomaly, the spatial coordinate range, and the confidence level identifier, a monitoring and early warning instruction containing spatiotemporal location information and severity information is generated.

10. A network camera monitoring and recognition system based on artificial intelligence, characterized in that, The device includes a processor and a memory, the memory being connected to the processor. The memory is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the memory to implement the artificial intelligence-based network camera monitoring and recognition method according to any one of claims 1-9.

Images