Monitoring video AI analysis system based on computer vision and deep learning technology

Abstract

The invention discloses a monitoring video AI analysis system based on computer vision and a deep learning technology, and aims to solve the problems of high false detection rate of dynamic background interference and high omission rate of small targets of an existing system. The system comprises a video data preprocessing module, a dynamic background adaptive suppression module, a small target enhancement analysis module and an AI analysis result output and feedback module. Wherein the dynamic background adaptive suppression module automatically distinguishes a dynamic background from a foreground target through space-time attention and residual learning, and outputs a foreground candidate frame; the small target enhancement analysis module extracts multi-scale features based on improved ResNet-18, enhances small target features in combination with hierarchical attention weighted fusion, and realizes small target detection and behavior analysis through improved YOLOv8. The method can be widely applied to the fields of security, traffic and the like.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and deep learning, specifically to an AI analysis system for surveillance videos based on computer vision and deep learning technologies. Background Technology

[0002] Existing AI analysis systems for surveillance videos largely rely on traditional object detection algorithms (such as the YOLO series and Faster R-CNN) and fixed background modeling methods (such as Gaussian mixture model MOG2 and frame difference methods), which have the following core problems: Dynamic background interference problem: When there are dynamic background elements such as swaying leaves, raindrops, and sudden changes in light in the monitoring scene, traditional background modeling methods cannot effectively distinguish between "background interference" and "foreground targets", resulting in a significant increase in the false detection rate (such as misjudging swaying leaves as pedestrians). Insufficient small target detection capability: distant targets in surveillance videos (such as pedestrians on the road or packages in the park) have small pixel areas and few feature information. Traditional multi-scale feature fusion is only achieved by simple downsampling and upsampling, which cannot effectively extract the key features of small targets, resulting in a false detection rate of more than 30%. Poor adaptability: Existing systems require manual parameter adjustment to adapt to different scenarios (such as day / night, sunny / rainy days), lacking adaptive optimization capabilities and making it difficult to meet the real-time analysis needs of complex monitoring environments.

[0003] To address the aforementioned issues, a surveillance video AI analysis system based on computer vision and deep learning technologies was designed. This system overcomes existing technological bottlenecks by focusing on dynamic background suppression and small target feature enhancement, thereby improving the accuracy and robustness of surveillance video analysis. Summary of the Invention

[0004] The purpose of this invention is to provide a surveillance video AI analysis system based on computer vision and deep learning technologies to solve the problems mentioned in the background.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a surveillance video AI analysis system based on computer vision and deep learning technologies, including a video data preprocessing module, a dynamic background adaptive suppression module, a small target enhancement analysis module, and an AI analysis result output and feedback module. The video data preprocessing module is used to denoise, normalize brightness, and crop frame sequences of the original surveillance video stream, outputting standardized 5-frame continuous data units to eliminate differences in the original data and provide high-quality input for subsequent modules. The dynamic background adaptive suppression module automatically distinguishes dynamic backgrounds such as swaying leaves and raindrops from foreground targets through spatiotemporal attention and residual learning, suppresses background interference, and outputs candidate frames containing only suspected foregrounds, thus solving the problem of high false detection rate in traditional methods. The small target enhancement analysis module extracts multi-scale features by improving ResNet-18, combines hierarchical attention weighted fusion to enhance small target features, and then uses improved YOLOv8 to realize the detection, classification and behavior analysis of small targets, thus solving the problem of high false negative rate of small targets; The AI ​​analysis result output and feedback module receives the output results from the small target enhancement analysis module, including target category, target location, and behavior status; stores the results in the database and pushes them to the monitoring terminal through the API interface; at the same time, it optimizes the attention weight parameters of the small target enhancement module based on historical data feedback to improve long-term adaptability.

[0006] Preferably, the specific implementation steps of the dynamic background adaptive suppression module are as follows: Step 1: Preprocess the continuous (like A video sequence of frames is divided into spatial dimensions. The spatiotemporal blocks are divided into M=16, meaning each spatiotemporal block contains 16×16×5 pixel data, where 5 represents the number of frames in the time dimension; a 3-layer convolutional neural network is used to extract the spatiotemporal features of each spatiotemporal block. The formula is: in, The raw data for a single spatiotemporal block. The convolutional kernels of the three CNN layers are respectively 3×3×5, 3×3×64, and 3×3×128. For the bias term of the corresponding layer, For activation function, The extracted spatiotemporal feature matrix; Step 2, Spatiotemporal Attention Weight Calculation: Design a spatiotemporal attention mechanism and calculate the attention weight for each spatiotemporal block. Used to distinguish between "background blocks" and "foreground blocks": in, Spatiotemporal characteristics Height, width, number of channels; For the first The characteristic mean of each channel (reflecting the background characteristic benchmark of that channel); Representing spatiotemporal characteristics The height dimension index, with values ​​ranging from 1 to... ; Representing spatiotemporal characteristics The index of the width dimension has a value range of 1 to... ; An adaptive threshold (trained from the first 100 frames of background data, initial value) ); The closer the value is to 0, the higher the probability that the spatiotemporal block is background; the closer it is to 1, the higher the probability that it is foreground. Step 3, Residual Background Suppression and Foreground Candidate Frame Generation: Incorporating spatiotemporal features... With attention weight Element-wise multiplication suppresses background features while preserving foreground features; a residual term is also introduced. (The residual network output obtained from training on the background frame) further eliminates background interference, as shown in the formula: in, Foreground feature matrix, This is the residual background suppression term (learned from pure background video frames in the training set, used to counteract the residual effects of background features); it represents the sum of all spatiotemporal blocks. By stitching the frames together and then performing a deconvolution operation to restore the original frame resolution, foreground candidate frames are obtained. (The background interference is significantly suppressed in areas containing only suspected foreground targets).

[0007] Preferably, in step 2 Adaptive threshold The method to obtain it is as follows: Step 2.1: The video data preprocessing module first acquires 100 frames of pure background video data without any foreground targets. Specific requirements are as follows: Acquisition scene: It must be completely consistent with the scene to be monitored and analyzed later (such as a school playground or city road) to ensure that background elements (such as leaves, streetlights, and ground textures) match the actual operating environment; Acquisition duration: The monitoring video frame rate is 25fps, and the acquisition duration for 100 frames is 4 seconds to avoid incomplete background features due to insufficient acquisition time; Data filtering: Frames containing foreground targets (such as pedestrians and vehicles) are filtered through manual annotation or a simple frame difference method (difference between adjacent frames is less than 5%) to ensure that all 100 frames are pure background frames without any foreground interference. Step 2.2, Pure Background Data Preprocessing: The video data preprocessing module preprocesses the 100 frames of pure background video data without foreground targets obtained in Step 2.1. The preprocessing includes Gaussian filtering for noise reduction, brightness normalization, and frame sequence cropping. The frame sequence cropping is as follows: The time window divides 100 frames into 20 groups of consecutive frame sequences, i.e., 5 frames per group, with each group serving as a spatiotemporal feature extraction unit. Step 2.3, Spatiotemporal Feature Extraction of Pure Background: For each group of 5 frames of pure background sequence, perform the same spatiotemporal feature extraction operation as in Step 1, specifically: Spatiotemporal block partitioning: Each group of 5-frame sequences (resolution calculated at 1080P, i.e., 1920×1080×5) is divided into... The spatiotemporal blocks are 16×16×5 in size (16×16 pixels in space, 5 frames in time). Each sequence generates (1920 / 16)×(1080 / 16)=120×67.5→120×67=8040 spatiotemporal blocks (blocks with edges less than 16×16 are zero-padded). 3-layer CNN feature extraction: For each pure background spatiotemporal block, the feature extraction is performed using the formula: Extracting background features and spatiotemporal features , among which, in this formula This is pure background spatiotemporal block data. ; Step 2.4: Background Feature Benchmark Calculation and Threshold Learning: All spatiotemporal block features based on 100 frames of pure background Calculate background feature benchmark And further learn adaptive threshold Specifically: Calculate background feature benchmark Features of all pure background spatiotemporal blocks The mean is calculated based on the channel dimension (C=128), i.e. ,in The total number of spatiotemporal blocks for 100 frames of pure background data (20 sets of sequences × 8040 blocks / set = 160800 blocks). For the first The pure background spacetime block in the first The characteristic values ​​of each channel, (Each channel corresponds to a background reference value); Calculate the background feature deviation distribution: For each pure background spatiotemporal block, calculate its distribution with the background feature deviation distribution according to the formula above. The mean channel deviation, i.e. , For the first Background deviation value of a pure background spatiotemporal block; express The position index of the height dimension has a value range of 1 to 1. ; express The position index of the width dimension ranges from 1 to... ; Step 2.5, Learning Threshold For all pure background spacetime blocks Statistical analysis was performed, and its 95th percentile was calculated as the initial threshold. .

[0008] Preferably, in step 3, the residual term The specific method for obtaining it is as follows: Step 3.1: Construction of a Diverse Pure Background Dataset: First, construct a pure background dataset covering the target monitoring scene. Specific requirements are as follows: Data Scene: Includes all typical background interference types of the target monitoring scene. For example, a school playground scene should include sub-scenes such as "leaf swaying (windy day), ground shadow changes (sunny afternoon), and light flickering (night)"; Data Volume: Collect 1000 frames of pure background data for each sub-scene, totaling 5-10 sub-scenes (to ensure generalization), with a total data volume of 5000-10000 frames; Data Format: All frames are preprocessed through the video data preprocessing module (Gaussian filtering, brightness normalization) and divided into 16×16×5 spatiotemporal blocks (consistent with the input format mentioned above). The final pure background spatiotemporal block dataset should have a size of 100,000-200,000 (to ensure sufficient training).

[0009] Step 3.2, Residual Network Training: Network architecture design: The residual network contains 3 residual blocks, each consisting of "2 layers of 3×3 convolutions + ReLU activation + shortcut connections", with an input dimension of 16×16×128 (and...). (Consistent), the output dimension is also 16×16×128 (and) (The dimensions must be consistent). Training objective function: using mean squared error loss, the objective is the network output. With input satisfy The loss function formula is: ,in The number of pure background spatiotemporal blocks in the training set. For the network to the first indivual The output residual terms; Training Process: Optimizer: The Adam optimizer is used, with an initial learning rate of 1e-4, which decays to 0.5 every 10 epochs; Number of Iterations: Training is conducted for 50 epochs until the loss function converges (loss fluctuation is less than 1e-5 for 5 consecutive epochs); Model Saving: After training, the weight parameters of the residual network are saved. During subsequent system runtime, any pure background spatiotemporal block is input. The corresponding output can then be obtained through this network. .

[0010] Preferably, the specific implementation logic of the small target enhancement analysis module is as follows: Step A: Multi-scale feature hierarchical extraction: using foreground candidate frames output by the dynamic background suppression module. As input, an improved ResNet-18 network (removing the last two fully connected layers and retaining the first five convolutional layers) is used to extract features at three scales: Low-scale features This is from the second convolutional layer, with a resolution of 1 / 4 of the original frame and 64 channels (containing detailed features such as edges and textures of small objects). Mesoscale features : From the 3rd convolutional layer, with a resolution of 1 / 8 of the original frame and 128 channels (containing local shape features of small objects). High-scale features : From the 4th convolutional layer, with a resolution of 1 / 16 of the original frame and 256 channels (containing semantic category features of small objects). Step B: Hierarchical Attention Weight Calculation: A hierarchical attention mechanism is designed to calculate the attention weights at each scale, taking into account the contribution of features at different scales to the small target. ,in : in, For the first The "significance index" of features at each scale. For the first Each scale feature The channel variance of the location (the larger the variance, the higher the probability that the location contains small target features); , The first The height and width of each scale feature; These represent the attention weights for low-, medium-, and high-scale features, with the latter being particularly relevant for small-object scenarios. Weighting ≥ 50% (enhancing detailed features); Step C, Feature Fusion and Small Target Reconstruction: Features at various scales are upsampled, i.e., bilinear interpolation is used to unify them to low-scale features. resolution, The resolution is 1 / 4 of the original frame, and the data is weighted and fused according to attention weights: in, For upsampling operation, For the fused feature matrix (the number of channels is unified to 256 through 1×1 convolution); a "small target feature reconstruction module" is designed, which connects to the residual through 3×3 convolution to reconstruct the feature matrix. The small target features are enhanced using the following formula: in, It uses a 3×3 convolution kernel (256 channels). For bias terms, The enhanced feature matrix of small targets; Step D, Small Target Detection and Classification: Input is fed into an improved YOLOv8 detector head (reducing downsampling times and improving adaptability to small targets), and outputs the position of the small target (bounding box). The system includes categories (such as pedestrians, vehicles, and abnormal objects) and confidence levels (with a threshold of 0.5 to filter low-confidence detection results). It also combines continuous frame target tracking (using Kalman filtering) to achieve trajectory analysis and behavior judgment of small targets (such as whether there are abnormal behaviors such as wandering or crossing boundaries).

[0011] Compared with the prior art, the beneficial effects of the present invention are: Significant effect in suppressing dynamic background interference: This invention uses a dynamic background adaptive suppression module of "spatiotemporal attention + residual learning" to automatically distinguish dynamic backgrounds such as swaying leaves and rain from foreground targets. Compared with traditional fixed background modeling methods (such as MOG2 and frame difference method), the false detection rate is reduced in dynamic background scenes, avoiding misjudging background interference as foreground targets.

[0012] The small target detection capability is significantly improved: The small target enhancement analysis module of this invention uses an improved ResNet-18 to extract multi-scale features, combines hierarchical attention weighted fusion to enhance small target features, and then adapts the improved YOLOv8 detection head to meet the needs of small target detection. For small targets with a pixel area <64×64 (such as pedestrians in the distance or packages in the park), the detection rate and classification accuracy are improved, solving the problem of high false negative rate for small targets in traditional systems.

[0013] Strong scene adaptability: This invention has an adaptive threshold. Residual terms can be generated and updated periodically through learning from pure background data. It can adapt to different dynamic background scenarios through residual networks, and the AI ​​analysis result output and feedback module can also optimize the attention weight of small target modules based on historical data. No manual parameter adjustment is required to adapt to different monitoring scenarios such as day / night and sunny / rainy days, reducing manual maintenance costs.

[0014] Real-time performance meets practical needs: The computational complexity of the core module is controllable, and the single-frame processing time for dynamic background suppression and small target enhancement analysis is <40ms. It can adapt to the frame rate of conventional monitoring videos at 25fps and can output target category, location and behavior status in real time, meeting the real-time monitoring and analysis needs of scenarios such as security, traffic and park management. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall system structure of the present invention; Figure 2 This is a schematic diagram of the dynamic background adaptive suppression module of the present invention. Figure 3 This is a schematic diagram of the workflow of the small target enhancement analysis module of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1-3 This invention provides a technical solution: a surveillance video AI analysis system based on computer vision and deep learning technology, including a video data preprocessing module, a dynamic background adaptive suppression module, a small target enhancement analysis module, and an AI analysis result output and feedback module. The video data preprocessing module is used to denoise, normalize brightness, and crop frame sequences of the original surveillance video stream, outputting standardized 5-frame continuous data units to eliminate differences in the original data and provide high-quality input for subsequent modules. The dynamic background adaptive suppression module automatically distinguishes dynamic backgrounds such as swaying leaves and raindrops from foreground targets through spatiotemporal attention and residual learning, suppressing background interference and outputting candidate frames containing only suspected foreground elements, thus solving the problem of high false detection rate in traditional methods. The specific implementation steps are as follows: Step 1: Preprocess the continuous (like A video sequence of frames is divided into spatial dimensions. The spatiotemporal blocks are divided into M=16, meaning each spatiotemporal block contains 16×16×5 pixel data, where 5 represents the number of frames in the time dimension; a 3-layer convolutional neural network is used to extract the spatiotemporal features of each spatiotemporal block. The formula is: in, The raw data for a single spatiotemporal block. The convolutional kernels of the three CNN layers are respectively 3×3×5, 3×3×64, and 3×3×128. For the bias term of the corresponding layer, For activation function, The extracted spatiotemporal feature matrix; Step 2, Spatiotemporal Attention Weight Calculation: Design a spatiotemporal attention mechanism and calculate the attention weight for each spatiotemporal block. Used to distinguish between "background blocks" and "foreground blocks": in, Spatiotemporal characteristics Height, width, number of channels; For the first The characteristic mean of each channel (reflecting the background characteristic benchmark of that channel); Representing spatiotemporal characteristics The height dimension index, with values ​​ranging from 1 to... ; Representing spatiotemporal characteristics The index of the width dimension has a value range of 1 to... ; An adaptive threshold (trained from the first 100 frames of background data, initial value) ); The closer the value is to 0, the higher the probability that the spatiotemporal block is background; the closer it is to 1, the higher the probability that it is foreground. Step 3, Residual Background Suppression and Foreground Candidate Frame Generation: Incorporating spatiotemporal features... With attention weight Element-wise multiplication suppresses background features while preserving foreground features; a residual term is also introduced. (The residual network output obtained from training on the background frame) further eliminates background interference, as shown in the formula: in, Foreground feature matrix, This is the residual background suppression term (learned from pure background video frames in the training set, used to counteract the residual effects of background features); it represents the sum of all spatiotemporal blocks. By stitching the frames together and then performing a deconvolution operation to restore the original frame resolution, foreground candidate frames are obtained. (Only areas containing suspected foreground targets are included, and background interference is significantly suppressed); Adaptive threshold and residual background suppression term These are the core parameters of the dynamic background adaptive suppression module. Both are constructed based on "pure background data learning" and need to be deeply coupled with the module's spatiotemporal feature extraction logic to ensure the accuracy of subsequent background suppression. The following section, in conjunction with the overall invention process, details the methods for obtaining these two parameters step by step, while also clarifying their connection to the modules mentioned earlier: The core function of the adaptive threshold T is to distinguish the feature difference threshold between "background spatiotemporal blocks" and "foreground spatiotemporal blocks" in the calculation of spatiotemporal attention weights—when the deviation of the features of a spatiotemporal block from the background baseline is less than a certain threshold, the threshold value is determined by the difference between the features of a spatiotemporal block and the background baseline. When this is the case, it is judged as background (attention weight). If the value approaches 0, it is considered a potential prospect (or close to 0). Approaching 1). Its acquisition requires four steps: "background data acquisition → data preprocessing → feature benchmark calculation → threshold learning", and the entire process is consistent with the module's spatiotemporal block division and feature extraction logic.

[0018] Adaptive threshold The method to obtain it is as follows: Step 2.1: The video data preprocessing module first acquires 100 frames of pure background video data without any foreground targets. Specific requirements are as follows: Acquisition scene: It must be completely consistent with the scene to be monitored and analyzed later (such as a school playground or city road) to ensure that background elements (such as leaves, streetlights, and ground textures) match the actual operating environment; Acquisition duration: The monitoring video frame rate is 25fps, and the acquisition duration for 100 frames is 4 seconds to avoid incomplete background features due to insufficient acquisition time; Data filtering: Frames containing foreground targets (such as pedestrians and vehicles) are filtered through manual annotation or a simple frame difference method (difference between adjacent frames is less than 5%) to ensure that all 100 frames are pure background frames without any foreground interference. Step 2.2, Pure Background Data Preprocessing: The video data preprocessing module preprocesses the 100 frames of pure background video data without foreground targets obtained in Step 2.1. The preprocessing includes Gaussian filtering for noise reduction, brightness normalization, and frame sequence cropping, specifically: Gaussian filtering for noise reduction: using 5×5 cores, Gaussian filtering is used to eliminate high-frequency noise in the background (such as light spots). Brightness normalization: Mapping pixel values ​​to The interval is used to avoid the impact of light fluctuations on feature extraction; Frame sequence cropping: as described above The time window divides 100 frames into 20 consecutive frame sequences (5 frames per group), with each group serving as a spatiotemporal feature extraction unit (consistent with the input format of spatiotemporal block division described later). Step 2.3, Spatiotemporal Feature Extraction of Pure Background: For each group of 5 frames of pure background sequence, perform the same spatiotemporal feature extraction operation as in Step 1, specifically: Spatiotemporal block partitioning: Each group of 5-frame sequences (resolution calculated at 1080P, i.e., 1920×1080×5) is divided into... The spatiotemporal blocks are 16×16×5 in size (16×16 pixels in space, 5 frames in time). Each sequence generates (1920 / 16)×(1080 / 16)=120×67.5→120×67=8040 spatiotemporal blocks (blocks with edges less than 16×16 are zero-padded). 3-layer CNN feature extraction: For each pure background spatiotemporal block, the feature extraction is performed using the formula: Extracting background features and spatiotemporal features , among which, in this formula This is pure background spatiotemporal block data. ; Step 2.4: Background Feature Benchmark Calculation and Threshold Learning: All spatiotemporal block features based on 100 frames of pure background Calculate background feature benchmark And further learn adaptive threshold Specifically: Calculate background feature benchmark Features of all pure background spatiotemporal blocks The mean is calculated based on the channel dimension (C=128), i.e. ,in The total number of spatiotemporal blocks for 100 frames of pure background data (20 sets of sequences × 8040 blocks / set = 160800 blocks). For the first The pure background spacetime block in the first The characteristic values ​​of each channel, (Each channel corresponds to a background reference value); Calculate the background feature deviation distribution: For each pure background spatiotemporal block, calculate its distribution with the background feature deviation distribution according to the formula above. The mean channel deviation, i.e. , For the first Background deviation value of a pure background spatiotemporal block; express The position index of the height dimension has a value range of 1 to 1. ; express The position index of the width dimension ranges from 1 to... ; Step 2.5, Learning Threshold For all pure background spacetime blocks Statistical analysis was performed, and its 95th percentile was calculated as the initial threshold. The 95th percentile was chosen because it covers the normal deviation of 95% of pure background blocks (avoiding misidentification of background as foreground) while identifying abnormal deviations exceeding this quantile (potentially introduced by foreground objects) as suspected foreground, thus aligning with the core objective of "suppressing background and preserving foreground" mentioned earlier. For example, 100 frames of pure background data from a school playground. After statistical analysis, the 95th percentile is 0.8, which is the initial value. ; To avoid environmental changes (such as gradual changes in lighting or minor changes in background elements) causing... Failure, the system generally needs to be updated periodically during operation. The specific mechanism is as follows: Update cycle: An update is triggered once per hour (which can be adjusted according to the complexity of the scene); Update data: Pure background frames without foreground targets are automatically selected from the monitoring frames of the past hour (judged by a trained background suppression module). (Frames identified as pure background) are accumulated to 100 frames. After this, steps 2.2-2.5 are repeated to update. Value; Update constraint: the value after each update The difference from the previous value should not exceed ±0.1 to avoid fluctuations in the false detection rate caused by sudden changes in the threshold.

[0019] Residual background suppression term Its core function is to offset the spatiotemporal attention weights in the foreground feature calculation (corresponding to step 3 above). Incompletely suppressed residual background features make It preserves foreground features more purely. Dimensions and spatiotemporal block features Completely identical (16×16×128), its acquisition requires two steps: "construction of pure background dataset → training of residual network".

[0020] residuals The specific method for obtaining it is as follows: Step 3.1: Constructing a Diverse Pure Background Dataset: To ensure that R can adapt to different dynamic background scenes (such as swaying leaves, raindrops, and sudden changes in light), a pure background dataset covering the target monitoring scene needs to be constructed. Specific requirements are as follows: Data scenarios: Include all typical background interference types of the target monitoring scenario. For example, a school playground scenario should include sub-scenarios such as "leaf swaying (windy day), ground shadow changes (sunny afternoon), and light flickering (night)". Data volume: 1000 frames of pure background data are collected for each sub-scene, totaling 5-10 sub-scenes (to ensure generalization), with a total data volume of 5000-10000 frames; Data format: All frames are preprocessed by the video data preprocessing module (Gaussian filtering, brightness normalization) and divided into 16×16×5 spatiotemporal blocks (consistent with the input format mentioned above). The final pure background spatiotemporal block dataset has a size of 100,000 to 200,000 (to ensure sufficient training).

[0021] Step 3.2, Residual Network Training: Network architecture design: The residual network contains 3 residual blocks, each consisting of "2 layers of 3×3 convolutions + ReLU activation + shortcut connections", with an input dimension of 16×16×128 (and...). (Consistent), the output dimension is also 16×16×128 (and) (The dimensions must be consistent). Training objective function: using mean squared error loss, the objective is the network output. With input satisfy The loss function formula is: ,in The number of pure background spatiotemporal blocks in the training set. For the network to the first indivual The output residual terms; Training Process: Optimizer: The Adam optimizer is used, with an initial learning rate of 1e-4, which decays to 0.5 every 10 epochs; Number of Iterations: Training is conducted for 50 epochs until the loss function converges (loss fluctuation is less than 1e-5 for 5 consecutive epochs); Model Saving: After training, the weight parameters of the residual network are saved. During subsequent system runtime, any pure background spatiotemporal block is input. The corresponding output can then be obtained through this network. .

[0022] During system operation, It needs to be compared with the spatiotemporal block features currently being processed. Real-time matching, the specific adaptation logic is as follows: Real-time generation: Based on the current spatiotemporal block features extracted in step 3.1 Firstly, through " Preliminary judgment" - if (If determined to be a high-probability background block), then... Input the trained residual network and output the corresponding spatiotemporal block. ;like (If determined to be a suspected foreground block), then... Set it to a zero matrix (to avoid canceling out foreground features); Scene adaptation: If the monitored scene undergoes a sudden change (e.g., from sunny to rainy), the system automatically triggers "residual network fine-tuning"—collecting 500 frames of pure background data in the new scene, and fine-tuning the network parameters according to the training process in step 3.2 (iterwise only 10 epochs) to ensure... Able to adapt to background features of new scenarios; dimensional consistency guaranteed. Spatial dimensions (16×16), number of channels (128), and current spatiotemporal block characteristics Completely consistent, ensuring subsequent Element-wise operations can be performed normally (avoiding dimension mismatch errors).

[0023] The small target enhancement analysis module improves ResNet-18 to extract multi-scale features, combines hierarchical attention weighted fusion to enhance small target features, and then uses an improved YOLOv8 to achieve small target detection, classification, and behavior analysis, thus solving the problem of high false negative rate for small targets. The specific implementation logic is as follows: Step A: Multi-scale feature hierarchical extraction: using foreground candidate frames output by the dynamic background suppression module. As input, an improved ResNet-18 network (removing the last two fully connected layers and retaining the first five convolutional layers) is used to extract features at three scales: Low-scale features This is from the second convolutional layer, with a resolution of 1 / 4 of the original frame and 64 channels (containing detailed features such as edges and textures of small objects). Mesoscale features : From the 3rd convolutional layer, with a resolution of 1 / 8 of the original frame and 128 channels (containing local shape features of small objects). High-scale features : From the 4th convolutional layer, with a resolution of 1 / 16 of the original frame and 256 channels (containing semantic category features of small objects). It should be noted that the specific improvement of "removing the last two fully connected layers and retaining the first five convolutional layers" ensures that the network structure is compatible with the original ResNet-18 structure, while also adapting to... The improved network retains only the initial convolutional layer + 4 residual blocks (including convolutional layers), for a total of 5 convolutional layers (corresponding to the definition of the "first 5 convolutional layers"); specifically: Structural pruning: The original ResNet-18 contains "initial convolutional layer + 4 residual blocks + average pooling layer + 2 fully connected layers". This invention removes "average pooling layer + 2 fully connected layers" and only retains the first 5 convolutional related structures (layer 1 to layer 5) to avoid the destruction of spatial features by fully connected layers (small target features depend on spatial location information). Channel and stride adaptation: The initial convolutional layer stride is set to 2 (instead of 1), which reduces computational cost while improving the 1080P resolution. The image is quickly compressed to 540×270. Subsequent residual blocks are downsampled stepwise with stride=2 to form multi-scale resolution differences. Residual connection preservation: All residual blocks retain the original ResNet's "shortcut connection" (the dimension is adjusted by 1×1 convolution when the number of channels changes), which solves the gradient vanishing problem of deep convolutions and ensures the effective extraction of high-level semantic features (layer 5).

[0024] Step B: Hierarchical Attention Weight Calculation: A hierarchical attention mechanism is designed to calculate the attention weights at each scale, taking into account the contribution of features at different scales to the small target. ,in : in, For the first The "significance index" of features at each scale. For the first Each scale feature The channel variance of the location (the larger the variance, the higher the probability that the location contains small target features); , The first The height and width of each scale feature; These represent the attention weights for low-, medium-, and high-scale features, with the latter being particularly relevant for small-object scenarios. Weighting ≥ 50% (enhancing detailed features); Step C, Feature Fusion and Small Target Reconstruction: Features at various scales are upsampled, i.e., bilinear interpolation is used to unify them to low-scale features. resolution, The resolution is 1 / 4 of the original frame, and the data is weighted and fused according to attention weights: in, For upsampling operation, For the fused feature matrix (the number of channels is unified to 256 through 1×1 convolution); a "small target feature reconstruction module" is designed, which connects to the residual through 3×3 convolution to reconstruct the feature matrix. The small target features are enhanced using the following formula: in, It uses a 3×3 convolution kernel (256 channels). For bias terms, The enhanced feature matrix of small targets; Step D, Small Target Detection and Classification: Input is fed into an improved YOLOv8 detector head (reducing downsampling times and improving adaptability to small targets), and outputs the position of the small target (bounding box). The system includes categories (such as pedestrians, vehicles, and abnormal objects) and confidence levels (with a threshold of 0.5 to filter low-confidence detection results). It also combines continuous frame target tracking (using Kalman filtering) to achieve trajectory analysis and behavior judgment of small targets (such as whether there are abnormal behaviors such as wandering or crossing boundaries).

[0025] The AI ​​analysis results output and feedback module receives the output results from the small target enhancement analysis module, including target category, target location, and behavior status; stores the results in the database and pushes them to the monitoring terminal via API interface; at the same time, it optimizes the attention weight parameters of the small target enhancement module based on historical data feedback to improve long-term adaptability.

[0026] The workflow of the AI ​​analysis results output and feedback module follows existing conventional technology, which will be briefly explained here: 1. Receive the output results from the small target module: Accurately receive structured data from the small target augmentation analysis module to ensure information integrity—the data includes three core categories: target category (pre-defined category labels such as pedestrians, vehicles, packages, etc.), target location (bounding box). Coordinates, and foreground candidate frames The system includes resolution matching, behavioral status (judgment results such as stationary, moving, and abnormal wandering based on Kalman filter trajectory analysis), along with result confidence (e.g., 0.85) and frame timestamps (accurate to milliseconds), providing a basis for subsequent storage and tracing.

[0027] 2. Results Storage and Structured Management: Received target information is stored in a MySQL database according to a standardized format to ensure efficient data retrieval. A composite primary key of "timestamp + monitoring device ID" is used to construct the data table structure, which includes six fields: timestamp, device ID, target category, bounding box coordinates, behavior status, and confidence level. During storage, the corresponding monitoring scene tags (such as school playground or city road) are automatically associated, laying the foundation for subsequent analysis of historical data and parameter optimization by scene.

[0028] 3. Terminal Visualization Push and Anomaly Warning: Results are pushed to various monitoring terminals via API interface to achieve real-time information delivery. For routine target information (such as pedestrians moving normally), it is displayed on the large screen as "target box + category label" superimposed on the real-time video stream, and key information is displayed in list form on the mobile APP (such as "2 pedestrians detected in the playground area at 14:30"). If abnormal behavior is detected (such as abnormal loitering), a tiered warning is triggered: a pop-up prompt on the large screen, a push sound and light reminder on the APP, and a video clip of the abnormal behavior (10 seconds before and after) is simultaneously sent to the terminal for quick verification.

[0029] 4. Historical Data Feedback and Parameter Optimization: Based on historical data in the database, the attention weight parameters of the small target module are dynamically optimized to form a closed-loop system iteration. Detection data for different scenarios (such as the detection rate and classification accuracy of small targets in a school playground scenario) are statistically analyzed weekly. If the detection rate of small targets (pixels < 64×64) in a certain scenario is lower than a preset threshold (e.g., 85%), then the low-scale features of that scenario are analyzed. Attention weights Is it insufficient? By calculating " "Adjusted detection rate increase" The initial proportion (e.g., 50%) is fine-tuned to the optimal value (e.g., 55%) and fed back to the small target enhancement analysis module of multi-scale feature fusion to improve the long-term adaptability of the system to the scenario.

[0030] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A monitoring video AI analysis system based on computer vision and deep learning technology, characterized in that, The video data preprocessing module is used for denoising, brightness normalization and frame sequence cropping on the original monitoring video stream, and outputs standardized 5-frame continuous data units, eliminating the differences between original data and providing high-quality input for subsequent modules. The dynamic background adaptive suppression module automatically distinguishes dynamic backgrounds such as leaf shaking and rain from foreground targets through spatio-temporal attention and residual learning, suppresses background interference, and outputs candidate frames containing only suspected foreground, solving the problem of high false detection rate of traditional methods. The small target enhancement analysis module extracts multi-scale features by improving ResNet-18, combines hierarchical attention weighted fusion to strengthen small target features, and then realizes small target detection, classification and behavior analysis through improved YOLOv8, solving the problem of high small target missed detection rate. The AI analysis result output and feedback module receives the output results of the small target enhancement analysis module, including target category, target position and behavior state. The results are stored in the database and pushed to the monitoring terminal through the API interface; at the same time, the attention weight parameters of the small target enhancement module are optimized according to the historical data feedback to improve the long-term adaptability. 2.The computer vision and deep learning based surveillance video AI analysis system according to claim 1, characterized in that: The dynamic background adaptive suppression module specifically implements the following steps: Step 1, for the pre-processed continuous frame video sequence, divide it into spatio-temporal blocks in spatial dimension; extract spatio-temporal features of each spatio-temporal block using a 3-layer convolutional neural network , formula: wherein, is the original data of a single spatio-temporal block, is the convolution kernel of a 3-layer CNN respectively, is the bias term of the corresponding layer, is the activation function, is the extracted spatio-temporal feature matrix; Step 2, Spatio-temporal attention weight calculation: design a spatio-temporal attention mechanism to calculate the attention weight of each spatio-temporal block , for distinguishing "background blocks" from "foreground blocks": wherein, respectively are height, width, channel number of the spatio-temporal feature ; is the feature mean of the th channel; is the adaptive threshold value; represents the index of the height dimension of the spatio-temporal feature , and the value range is 1 to ; represents the index of the width dimension of the spatio-temporal feature , and the value range is 1 to ; , the value closer to 0 indicates that the probability of the spatio-temporal block being background is higher, and closer to 1 indicates that the probability of being a foreground target is higher; Step 3, residual background suppression and foreground candidate frame generation: multiply the spatio-temporal features with attention weights element by element, suppress the features of background blocks, and retain the features of foreground blocks; meanwhile, introduce a residual term to further eliminate background interference, and the formula is: wherein, is the foreground feature matrix, is the residual background suppression term; concatenating all spatio-temporal blocks patched, restored to the original frame resolution by an inverse convolution operation, resulting in a foreground candidate frame . 3.The computer vision and deep learning based surveillance video AI analysis system according to claim 2, characterized in that: The step 2 in Adaptive threshold The acquisition method is as follows: Step 2.1, the video data preprocessing module first acquires 100 frames of pure background video data without foreground targets, with the following specific requirements: scene collection: completely consistent with the scene of the monitoring analysis acquired by the video data preprocessing module, ensuring that the background elements match the actual operating environment; collection time: 25fps frame rate of monitoring video, 100 frames corresponding to 4 seconds of collection time, avoiding too short collection time leading to incomplete background features; Data screening: filter frames containing foreground targets through manual annotation or simple frame difference method to ensure that 100 frames are pure background frames without any foreground interference; Step 2.2, pure background data preprocessing: the video data preprocessing module pre-processes the 100 frames of pure background video data without foreground objects obtained in step 2.1, wherein the pre-processing includes Gaussian filter denoising, brightness normalization and frame sequence cropping, wherein the frame sequence cropping is to divide the 100 frames into 20 groups of continuous frame sequences, i.e. 5 frames per group, and each group is taken as a spatio-temporal feature extraction unit. Step 2.3, pure background data feature extraction: the video data feature extraction module extracts the spatio-temporal features of the 20 groups of continuous frame sequences obtained in step 2.2, wherein the spatio-temporal features include the following features: Step 2.3, pure background spatio-temporal feature extraction: for each group of 5-frame pure background sequence, perform the same spatio-temporal feature extraction operation as step 1, specifically: Temporal-spatial block division: each group of 5 frame sequences is divided into temporal-spatial blocks, each temporal-spatial block has a size of 16x16x5, and each group of sequences generates (1920 / 16)x(1080 / 16)=120x67.5→120x67=8040 temporal-spatial blocks; 3-layer CNN feature extraction: for each pure background spatiotemporal block, the background feature spatiotemporal feature is extracted by formula: wherein, in the formula, is the pure background spatiotemporal block data, ;​ Step 2.4, background feature baseline calculation and threshold Learning: based on 100 frames of pure background, all spatio-temporal block features , calculate the background feature baseline , and further learn adaptive threshold , specifically: Computing background feature reference : features of all pure background spatio-temporal blocks , compute mean value in channel dimension, i.e. , where is the total number of spatio-temporal blocks of 100 frames of pure background data, is the feature value of the th pure background spatio-temporal block in the th channel, ; Compute background feature bias distribution: for each pure background spatiotemporal block, compute its channel bias mean with the formula in the previous section, i.e. , , is the background-in bias value of the th pure background spatiotemporal block; denotes the position index in the height dimension, whose value range is from 1 to ; denotes the position index in the width dimension, whose value range is from 1 to ;​​​ Step 2.5, learning threshold : Statistics are performed on all pure background spatiotemporal blocks and the 95% quantile is computed as the initial threshold . 4.The computer vision and deep learning based surveillance video AI analysis system according to claim 2, characterized in that: The residual term in step 3 The specific acquisition method is as follows: Step 3.1, diversified pure background dataset construction: first construct a pure background dataset covering the target monitoring scene, with the following specific requirements: data scene: contains all typical background interference types of the target monitoring scene; Data volume: 1000 frames of pure background data are collected for each sub-scene, totaling 5-10 sub-scenes, with a total data volume of 5000-10000 frames; Data Format: all frames are preprocessed by the video data preprocessing module and divided into 16x16x5 spatio-temporal blocks, and the final pure background spatio-temporal block dataset is generated. Step 3.2, residual network training: Network structure design: the residual network contains 3 residual blocks, each composed of "2 layers of 3x3 convolution + ReLU activation + shortcut connection", with an input dimension of 16x16x128 and an output dimension of 16x16x128; Training objective function: using mean square error loss, the goal is the output of the network With input Satisfies , the loss function formula is , wherein The number of pure background space-time blocks in the training set, The residual term of the network output for the first Output​ Training process: Optimizer: Adam optimizer is adopted with initial learning rate of 1e-4, and the learning rate is reduced to 0.5 of the original every 10 epochs; iteration number: 50 epochs are trained until the loss function converges; model saving: after the training is completed, the weight parameters of the residual network are saved, and when the subsequent system runs, the corresponding can be output through the network. 5.The computer vision and deep learning based surveillance video AI analytics system of claim 1, wherein: The small target enhancement analysis module specifically implements the following logic: Step A, multi-scale feature hierarchical extraction: the foreground candidate frame output by the dynamic background suppression module For input, an improved ResNet-18 network is used to extract features of 3 scales: Low scale features : from the 2nd layer convolutional layer, resolution is 1 / 4 of the original frame, channel number 64; mesoscale features : from the 3rd layer of convolutional layers, resolution is 1 / 8 of the original frame, channel number is 128; high scale features : from the 4th layer convolutional layer, resolution is 1 / 16 of the original frame, channel number 256; Step B, hierarchical attention weight calculation: for the contribution of different scale features to small targets, a hierarchical attention mechanism is designed to calculate the attention weight of each scale wherein : wherein, is the "feature saliency indicator" of the th scale feature, is the channel variance of the th scale feature at the th position; , are the height and width of the th scale feature, respectively; are the attention weights of low, medium, and high scale features, respectively, wherein the weight ratio of is ≥ 50% in a small target scene. Step C, feature fusion and small target reconstruction: each scale feature is unified to the low scale feature by upsampling, that is, using bilinear interpolation with a resolution of 1 / 4 of the original frame, with a resolution of 1 / 4 of the original frame, and weighted fusion according to attention weights: wherein, is an up-sampling operation, is a fused feature matrix; a small target feature reconstruction module is designed, 3x3 convolution and residual connection are used to strengthen the small target feature in , and the formula is: wherein, is a 3x3 convolution kernel, is a bias term, is a small target feature matrix after enhancement; Step D, small target detection and classification: the input is input to the improved YOLOv8 detection head, and the position, category and confidence of the small target are output; at the same time, the trajectory analysis and behavior judgment of the small target are realized by combining the target tracking of the continuous frames. Step D, small target detection and classification: the input is input to the improved YOLOv8 detection head, and the position, category and confidence of the small target are output; at the same time, the trajectory analysis and behavior judgment of the small target are realized by combining the target tracking of the continuous frames

Images