Real-time recognition of abnormal behavior in video monitoring based on deep learning and security warning system

Abstract

This invention discloses a real-time video surveillance abnormal behavior recognition and security early warning system based on deep learning, belonging to the field of intelligent video surveillance technology. It solves the technical problems in existing technologies, such as the ease with which holes are generated in foreground extraction, the lack of deep coupling between geometric features and template matching in behavior recognition, and the fixed early warning threshold. The system includes: a Gaussian mixture and frame difference selective update module that outputs a stable foreground mask based on a motion history linked list; an improved Snake contour filling module that introduces SSIM dynamic weights and displacement field convolution, combined with polar coordinate DCT filtering to output accurate contours; a dual-branch mutual attention behavior discrimination module, where the first branch encodes the attention prior vector through graph convolution, and the second branch performs latent space matching and backfeeds back to correct the threshold; a temporal memory verification module that sets a dynamic threshold based on confidence statistics and updates the template library online; and an early warning and storage linkage module that dynamically captures and uploads data to the cloud based on confidence inflection points.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video surveillance technology, specifically relating to a real-time identification and security early warning system for abnormal video surveillance behavior based on deep learning. Background Technology

[0002] With the increasing demand for public safety, real-time identification technology for abnormal behavior in intelligent video surveillance systems has become a research hotspot.

[0003] In existing technologies, abnormal behavior recognition typically includes moving target detection, behavior feature extraction, and classification and early warning. However, the following technical bottlenecks still exist: First, when extracting the foreground of a moving target using the traditional Gaussian mixture model combined with frame difference method, "holes" are easily generated inside the moving target. This leads to discontinuities and incompleteness in the target contours upon which subsequent behavior recognition relies. Existing methods often prevent "holes" by adjusting the update rate of different regions, but they cannot actively identify and repair already formed hole regions, making it difficult to output a stable foreground mask. Second, in existing behavior recognition methods, geometric feature discrimination based on centroid displacement and aspect ratio change rate, and discrimination based on template matching are often independent judgments or simple concatenations. Geometric features are difficult to distinguish complex abnormal behaviors, while template matching has a large computational load and is sensitive to pose changes. The lack of effective information interaction and mutual correction mechanisms between the two results in insufficient accuracy in the judgment of abnormal candidates. Third, most existing systems use fixed abnormal frame proportion thresholds or static matching confidence thresholds, which cannot be dynamically adjusted according to the distribution of normal behavior features and confidence fluctuations in the current scene. In complex environments such as changes in lighting and scene transitions, they are prone to missed detections or frequent false alarms.

[0004] Therefore, there is an urgent need for a video surveillance abnormal behavior real-time identification and security early warning system that can achieve complete foreground extraction, dual-branch mutual attention discrimination, and adaptive threshold warning. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art. To this end, this invention proposes a real-time video surveillance abnormal behavior recognition and security early warning system based on deep learning, which solves the technical problems in the prior art, such as the inability to actively identify and repair existing hole areas, the difficulty in outputting stable foreground masks, the lack of deep coupling between geometric features and template matching in behavior discrimination, and the fixed warning threshold and lack of scene adaptation capabilities.

[0006] To address the aforementioned problems, this invention provides a real-time video surveillance abnormal behavior identification and security early warning system based on deep learning, comprising the following modules: The Gaussian Mixture and Frame Difference Selective Update Module is used to model the background of video frames and extract the foreground region by fusing the frame difference method. It selectively updates the background model based on motion history information to output a stable foreground mask. Improved Snake contour filling module: The improved Snake contour filling module is used to extract the target contour based on the stable foreground mask, extract the linear fitting slope of the structural similarity index of consecutive frames in real time, define the descent rate interval according to the slope, and perform reverse weighted compensation on the foreground confidence constraint term according to the corresponding compensation intensity coefficient matched in the interval, and then fill the interior of the converged contour to generate a complete moving target image. The dual-branch mutual attention behavior discrimination module consists of two branches: a first branch, which calculates the temporal variation features of the target centroid displacement and the aspect ratio of the rectangle, and determines the target as an anomalous candidate if the value exceeds a threshold, and encodes the features as an attention prior vector; and a second branch, which receives the attention prior vector, performs adaptive template matching, outputs the matching confidence score of the current frame, and feeds the matching confidence score back to the first branch; the first branch uses the matching confidence score of the current frame to correct its dynamic threshold for geometric feature determination, and applies the corrected threshold to the anomalous candidate determination of the next frame; simultaneously, if the matching confidence score of the current frame is less than or equal to a preset anomalous confirmation threshold, the current frame is determined to be anomalous. Timing memory verification module: used to maintain the memory pool. When the proportion of abnormal frames is greater than the dynamic threshold, it outputs an early warning signal and uses the normal templates in the memory pool to update the template library of the second branch online. Early warning and storage linkage module: used to trigger an alarm and upload abnormal video clips to the cloud when an early warning signal is output.

[0007] Preferably, in the Gaussian mixture and frame difference selective update module, the selective update of the background model based on motion history information includes: A motion history linked list is established for each pixel to record the foreground determination state sequence of the pixel in past consecutive video frames; When there is a continuous segment of foreground states in the state sequence, the length of the continuous segment does not exceed 3 frames, and the frame before and the frame after the continuous segment are both background states, it is determined that the pixel is in a hole region inside the moving target, the Gaussian mixture model parameters corresponding to the pixel are rolled back to the historical parameter values ​​before the start of the continuous segment, and a forced background filling operation is performed on the pixel. When the length of the continuous segment of the foreground state in the state sequence increases frame by frame, the pixel is marked as the stable motion core area, and the background model parameter update of its area is paused. The foreground mask after rollback filling and pause update processing is output as a stable foreground mask.

[0008] Preferably, in the improved Snake contour filling module, extracting the target contour based on the stable foreground mask includes: Calculate the structural similarity index between the current frame and the current background model. The weighting coefficients of the prospect confidence constraint term Set as: The weighting coefficient of the internal energy term is set to... ,in, The preset maximum weight coefficient; Using the converged contour vertex coordinates of the previous frame as the initial position of the Snake model in the current frame, the normal direction displacement of each vertex of the contour in the previous frame is calculated, and a two-dimensional all-zero matrix with the same resolution as the current frame image is constructed. The normal direction displacement of each vertex is used as a pixel value and mapped to the corresponding vertex coordinate position in the two-dimensional all-zero matrix to generate a two-dimensional sparse displacement field matrix. The two-dimensional sparse displacement field matrix is ​​convolved with the gradient field of the two-dimensional image of the current frame by a size of 3*3 pixels to generate an initial velocity field that guides the evolution of the contour in the current frame. For 5 consecutive frames For video segments where the value decreases frame by frame, calculate the value within 5 frames. The slope of the linear fit of the value is used as the descent rate; the descent rate is divided into three intervals: low-speed, medium-speed, and high-speed, each corresponding to a different compensation intensity coefficient. The appropriate compensation intensity coefficient is selected based on the interval in which the descent rate falls. Perform reverse compensation, whereby, The weighting coefficients after compensation. The weighting coefficients before compensation. To compensate for the strength coefficient, The rate of decrease.

[0009] Preferably, in the improved Snake contour filling module, after filling the interior of the contour and generating a complete moving target image, a contour post-processing filtering step is also included: converting the converged contour curve to a polar coordinate system, arranging the polar radii in ascending order of polar angles with the geometric center of the contour as the pole; performing a discrete cosine transform on the polar radii sequence, retaining the first 12.5% ​​of the low-frequency coefficients, and then performing an inverse discrete cosine transform; combining the filtered polar radii sequence with the original polar angles, converting back to the vertex coordinates in the Cartesian coordinate system, and connecting each vertex to output the accurate target contour.

[0010] Preferably, in the dual-branch mutual attention behavior discrimination module, the first branch calculates the temporal variation features of the target centroid displacement and the aspect ratio of the rectangle and encodes the features into an attention prior vector, including: Extract the contour of the moving target from the stable foreground mask, calculate the geometric center of the contour as the centroid coordinates, and calculate the minimum bounding rectangle of the contour to obtain the width and height. The centroid coordinate sequence and width and height sequence are continuously acquired within L frames. Each frame is used as a graph node, and the centroid displacement and aspect ratio change rate between adjacent frames are used as edge features between nodes to construct a spatiotemporal graph. A graph convolutional network is used to extract features from the spatiotemporal graph. The centroid coordinates and aspect ratio of each frame are used as the initial feature vectors of the nodes, and the centroid displacement and aspect ratio change rate between adjacent frames are used as edge weights. The node information of adjacent frames and the node information of one frame apart are aggregated through three graph convolutional layers to generate the updated features of each node. Global average pooling is performed on the updated features of each node to obtain a graph-level feature vector, which is then output as the attention prior vector.

[0011] Preferably, the second branch receives the attention prior vector and performs adaptive template matching, including: Each standard behavior template in the template library contains a template feature vector and its corresponding binary image of the moving target. The template feature vector is generated by encoding the historical behavior sequence corresponding to the template through the same graph convolutional network and is in the same latent space as the attention prior vector. The attention prior vector is used to calculate the Euclidean distance with the template feature vector of each standard behavior template, and the top 3 templates with the smallest Euclidean distance are selected as the candidate template set. An affine transformation matrix is ​​constructed based on the centroid displacement component in the attention prior vector, and a rotation angle of is applied to the binary image of the moving target in the candidate template. The rotation transformation and scale factor are The scaling transformation, where, It is equal to the value of the angle of displacement of the center of mass. It is equal to the ratio of the effective bounding box area of ​​the target in the current frame to the bounding box area in the previous frame; Anisotropic scaling is performed on the candidate template based on the aspect ratio change rate component in the attention prior vector. The intersection-union ratio (IUR) of the aligned candidate template and the moving target image of the current frame is calculated pixel by pixel, and the maximum IUR is taken as the matching confidence output.

[0012] Preferably, the anisotropic scaling adjustment of the candidate template includes: Let the aspect ratio change rate component be... The scaling factor in the width direction is The scaling factor in the height direction is ; and satisfy: and Solving for the given information yields the following results. and ; Multiply the original width of the candidate template by After obtaining the adjusted width, multiply the original height of the candidate template by... After obtaining the adjusted height, perform independent resampling in the width and height directions on the binary image of the moving target in the candidate template to generate anisotropically scaled templates.

[0013] Preferably, the step of feeding back the matching confidence to the first branch to correct its threshold includes: Obtain the matching confidence of the second branch output. ,Establish Compared with the original threshold of the first branch Dynamic correction function between ,in, To preset the maximum correction step size, Sensitivity index; When matching confidence Three consecutive frames are greater than the lock threshold When the threshold correction operation is paused, the current threshold is maintained unchanged until two consecutive frames are passed. ≤ Then, restore the threshold correction; This serves as a new threshold for determining the first branch as an abnormal candidate.

[0014] Preferably, the time-series memory verification module includes: Maintain a circular memory pool of length N frames. Store the frame-level judgment result of each frame output by the dual-branch mutual attention behavior discrimination module into the circular memory pool. The frame-level judgment result includes an abnormal frame identifier and a matching confidence level. ; Count the number of abnormal frames marked as true in the circular memory pool, calculate the proportion of abnormal frames, and output an early warning signal when the proportion of abnormal frames is greater than the dynamic threshold; The dynamic threshold is calculated by extracting the matching confidence scores corresponding to normal frames in the circular memory pool. Sequence, calculation mean of the sequence with standard deviation ,set up: ,in, For dynamic thresholds, This is the sensitivity adjustment coefficient; After an early warning signal is output, a segment with more than K consecutive normal frames is extracted from the circular memory pool. The moving target images of each frame in the segment are weighted and fused to generate a new template. It is then determined whether the current template library capacity has reached its limit: if it has not reached the limit, it is directly expanded into the library; if it has reached the limit, the old template with the lowest usage frequency in the template library is replaced with the new template; when multiple old templates have the same lowest usage frequency, the old template with the earliest entry time is replaced according to the first-in-first-out principle.

[0015] Preferably, the early warning and storage linkage module includes: Maintain a circular video buffer to continuously store the original video frames and the corresponding moving target images; When the timing memory verification module outputs an early warning signal, it locates the continuous segment in the circular memory pool where the abnormal frame marker is true, and matches the confidence level within the continuous segment. The sequence is subjected to a one-dimensional moving average filter. The inflection points where the first derivative of the filtered sequence crosses zero and the second derivative is positive are calculated. If multiple inflection points exist, they are extracted. The global minimum inflection point with the smallest value and less than the preset absolute abnormal lower limit is used as the frame corresponding to the global minimum inflection point as the abnormal peak frame. Centered on the abnormal peak frame, based on the length of the abnormal continuous segment and minimum matching confidence Dynamically determine the truncation range: forward truncation length Backward truncation length ,in, and Preset positive coefficients; After the captured video clips are encoded and compressed, anomaly detection information is added to generate an alarm data packet, which is then uploaded to the cloud management platform. At the same time, the local alarm is triggered, and the alarm event is written to the local log database.

[0016] The beneficial effects of this invention are: This invention employs a technique of establishing a motion history linked list for each pixel to record the foreground determination state sequence. When the state sequence shows a continuous foreground state segment with a length not exceeding 3 frames and both preceding and following segments are background states, it is determined to be a hole region, and Gaussian mixture model parameter rollback and forced background filling are performed. When the foreground length in the state sequence shows a frame-by-frame increasing trend, it is determined to be a stable motion core region, and background model parameter updates are paused. This achieves direct repair of hole regions through parameter rollback and forced filling, and prevents the generation of new holes by pausing stable core region updates, resulting in a highly complete stable foreground mask. This invention constructs a first branch that uses a spatiotemporal graph combined with a graph convolutional network to encode the centroid displacement and aspect ratio change rate as an attention prior vector; it constructs a second branch that places the template feature vector and the attention prior vector in the same latent space, performs an affine transformation based on the centroid displacement component, performs anisotropic scaling based on the aspect ratio change rate component, and feeds back the matching confidence to the first branch to correct the threshold through a dual nonlinear continuous correction function. At the same time, a locking mechanism is set to prevent threshold oscillation, forming a mutual attention closed loop, which significantly improves the accuracy and robustness of abnormal candidate judgment. This invention constructs a circular memory pool to cache the anomaly identifier and matching confidence score of each frame, extracts the confidence score sequence of normal frames, and dynamically calculates the warning threshold using its mean and standard deviation. When the proportion of abnormal frames in the memory pool exceeds the threshold, an warning is triggered. At the same time, continuous normal frame segments are extracted and weighted to generate a new template and replace the old template with the lowest frequency of use in the template library. This achieves adaptive adjustment of the dynamic threshold according to the concentration and fluctuation of the confidence score of normal behavior, and the template library can evolve online, improving the accuracy of warnings and the adaptability of scenarios. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the module flow of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 As shown, this invention is a real-time video surveillance abnormal behavior identification and security early warning system based on deep learning, comprising the following modules: The Gaussian Mixture and Frame Difference Selective Update Module is used to model the background of video frames and extract the foreground region by fusing the frame difference method. It selectively updates the background model based on motion history information to output a stable foreground mask. Improved Snake contour filling module: The improved Snake contour filling module is used to extract the target contour based on the stable foreground mask, extract the linear fitting slope of the structural similarity index of consecutive frames in real time, define the descent rate interval according to the slope, and perform reverse weighted compensation on the foreground confidence constraint term according to the corresponding compensation intensity coefficient matched in the interval, and then fill the interior of the converged contour to generate a complete moving target image. The dual-branch mutual attention behavior discrimination module consists of two branches: a first branch, which calculates the temporal variation features of the target centroid displacement and the aspect ratio of the rectangle, and determines the target as an anomalous candidate if the value exceeds a threshold, and encodes the features as an attention prior vector; and a second branch, which receives the attention prior vector, performs adaptive template matching, outputs the matching confidence score of the current frame, and feeds the matching confidence score back to the first branch; the first branch uses the matching confidence score of the current frame to correct its dynamic threshold for geometric feature determination, and applies the corrected threshold to the anomalous candidate determination of the next frame; simultaneously, if the matching confidence score of the current frame is less than or equal to a preset anomalous confirmation threshold, the current frame is determined to be anomalous. Timing memory verification module: used to maintain the memory pool. When the proportion of abnormal frames is greater than the dynamic threshold, it outputs an early warning signal and uses the normal templates in the memory pool to update the template library of the second branch online. Early warning and storage linkage module: used to trigger an alarm and upload abnormal video clips to the cloud when an early warning signal is output.

[0020] In one embodiment of the present invention, the selective update module for Gaussian mixture and frame difference, which selectively updates the background model based on motion history information, includes: A motion history linked list is established for each pixel to record the foreground determination state sequence of the pixel in past consecutive video frames; When there is a continuous segment of foreground states in the state sequence, the length of the continuous segment does not exceed 3 frames, and the frame before and the frame after the continuous segment are both background states, it is determined that the pixel is in a hole region inside the moving target, the Gaussian mixture model parameters corresponding to the pixel are rolled back to the historical parameter values ​​before the start of the continuous segment, and a forced background filling operation is performed on the pixel. When the length of the continuous segment of the foreground state in the state sequence increases frame by frame, the pixel is marked as the stable motion core area, and the background model parameter update of its area is paused. The foreground mask after rollback filling and pause update processing is output as a stable foreground mask.

[0021] Specifically, for each input video frame, a Gaussian mixture model (Gaussian mixture model) is used to establish K Gaussian distributions for each pixel (K=3 in this embodiment) to describe the change of pixel values ​​over time. Simultaneously, inter-frame difference operations are performed on adjacent frames, and the difference results are logically ANDed with the foreground region extracted by the Gaussian mixture model to obtain the initial foreground region. Then, median filtering is used to remove isolated noise points, and shadow detection and removal are performed based on the HSV color space to obtain a preliminary foreground mask. A fixed-length circular queue is maintained for each pixel as a motion history linked list, with a queue length L of 20 frames. For each frame processed, the preliminary foreground determination result (1 for foreground, 0 for background) for that pixel in the current frame is pushed to the tail of the queue, and the last element at the head of the queue is popped. The earliest record shows that the linked list records the foreground / background state change trajectory of the pixel over the past 20 consecutive frames. A sliding window scan is performed on the motion history linked list of each pixel to detect the following state pattern: the length of a consecutive foreground state segment does not exceed 3 frames, and the frame preceding and following this segment is both background. The physical meaning of this pattern is: the pixel briefly jumps from the background to the foreground in the previous frame (lasting 1 to 3 frames), and then jumps back to the background. This "short-term transition" phenomenon is a typical spatiotemporal characteristic of small holes inside moving targets (such as gaps in clothing texture, gaps between arms and torsos). Real background pixels do not exhibit this brief foreground transition, while pixels inside real moving targets should remain in the foreground. If this pattern is detected... If the pattern is incorrect, the pixel is determined to be in a hole region inside the moving target. In this case, the following repair operations are performed: Locate the historical frame position before the start of the foreground continuous segment; roll back the Gaussian mixture model parameters (including the weights, mean, and variance of each Gaussian distribution) corresponding to the pixel to the parameter values ​​at that historical frame; and forcibly assign the pixel the background value (pixel value set to 0), directly filling the hole region. Through parameter rollback and forced filling, the hole region is no longer incorrectly regarded as background in subsequent frames, thus repairing the incompleteness of the foreground mask. For the motion history list of each pixel, it is checked whether the length of the foreground state continuous segment shows a frame-by-frame increasing trend. The specific judgment method is: take the foreground judgment results of the last 5 frames; if the length sequence of the foreground continuous segment (e.g., ...) shows an increasing trend, then the length of the foreground continuous segment is determined. If the length of frame t-4 is 2, the length of frame t-3 is 3, the length of frame t-2 is 4, the length of frame t-1 is 5, and the length of frame t is 6, and these values ​​show a monotonically increasing trend, then the pixel is determined to be in a stable motion core region (e.g., the central region of the human torso). At this point, the background model parameter update for this pixel is paused, meaning that the Gaussian model parameters for this pixel are not updated with a learning rate weight in the current frame, and the parameter values ​​from the previous frame remain unchanged. The purpose of this operation is to prevent regions with long-term motion from being incorrectly absorbed into the background model, thereby avoiding the creation of new "holes" inside the moving target. The foreground mask after hole repair and the pause of stable core region updates is used as the stable foreground mask output by this module for subsequent improvement of the Snake contour filling module.

[0022] In one embodiment of the present invention, the improved Snake contour filling module extracts the target contour based on the stable foreground mask, including: Calculate the structural similarity index between the current frame and the current background model. The weighting coefficients of the prospect confidence constraint term Set as: The weighting coefficient of the internal energy term is set to... ,in, The preset maximum weight coefficient; Using the converged contour vertex coordinates of the previous frame as the initial position of the Snake model in the current frame, the normal direction displacement of each vertex of the contour in the previous frame is calculated, and a two-dimensional all-zero matrix with the same resolution as the current frame image is constructed. The normal direction displacement of each vertex is used as a pixel value and mapped to the corresponding vertex coordinate position in the two-dimensional all-zero matrix to generate a two-dimensional sparse displacement field matrix. The two-dimensional sparse displacement field matrix is ​​convolved with the gradient field of the two-dimensional image of the current frame by a size of 3*3 pixels to generate an initial velocity field that guides the evolution of the contour in the current frame. For 5 consecutive frames For video segments where the value decreases frame by frame, calculate the value within 5 frames. The slope of the linear fit of the value is used as the descent rate; the descent rate is divided into three intervals: low-speed, medium-speed, and high-speed, each corresponding to a different compensation intensity coefficient. The appropriate compensation intensity coefficient is selected based on the interval in which the descent rate falls. Perform reverse compensation, whereby, The weighting coefficients after compensation. The weighting coefficients before compensation. To compensate for the strength coefficient, The rate of decrease.

[0023] Specifically, the coordinates of the converged contour vertices from the previous frame are used as the initial contour positions of the Snake model in the current frame. For the first frame, morphological edge detection is used to initialize the contour, and the structural similarity index between the current frame and the current background model is calculated. The calculation formula is as follows: ,in, For the current frame image, Background model image, The average pixel value of the current frame image. The pixel mean of the background model image. The pixel variance of the current frame image. The pixel variance of the background model image. The covariance between the current frame and the background model. It is the first stability constant. Let be the second stability constant, where , , This represents the dynamic range of pixel values ​​(255 for 8-bit images). , ; The value range is [0, 1]. The closer the value is to 1, the more similar the current frame is to the background model; the closer the value is to 0, the greater the scene change. The preset maximum weighting coefficient is set to 0.6 in this embodiment, and the weighting coefficient of the internal energy term is set to... , The closer to 1 The smaller the value, the more the contour evolution is dominated by internal energy to maintain smoothness; The closer to 0 The larger the value, the more dominant the foreground confidence constraint becomes in order to force the contour to fit the target boundary; The coordinates of each vertex of the converged contour in the previous frame are extracted, and the displacement of each vertex along the normal direction is calculated (i.e., the component of the difference between the current vertex position and its position in the previous frame in the normal direction). The normal displacements of all vertices are arranged in order on the contour to construct a displacement field matrix. This displacement field matrix is ​​then convolved with the gradient field of the current frame image, with a convolution kernel size of 3×3 pixels. The convolution result serves as the initial velocity field for the evolution of the Snake contour in the current frame, ensuring that the contour carries the motion inertia of the previous frame from the beginning of its evolution, thereby improving the temporal continuity of the contour between frames and avoiding contour jitter. The SSIM value of 5 consecutive frames is monitored. If the SSIM value decreases frame by frame within 5 frames (i.e., the SSIM of each frame is less than that of the previous frame), the linear fitting slope of the SSIM value within those 5 frames is calculated as the descent rate. , It is a positive value; The rate of decline The system is divided into three sections: low speed, medium speed, and high speed, as shown in Table 1.

[0024] Table 1 Among them, the above boundary values ​​(low-speed boundary 0.05, based on the fact that in normal monitoring scenarios, the SSIM decrease rate caused by slow changes in background illumination (sunrise and sunset) is usually less than 0.05. Below this value, the traditional Snake model can already track the contour well without significant intervention; high-speed boundary 0.15: when the target moves rapidly or the scene changes abruptly, the SSIM decrease rate between adjacent frames often exceeds 0.15. When this value is exceeded, the probability of contour divergence in the traditional Snake model increases significantly, requiring strong compensation), and their corresponding compensation intensity coefficients, including , and The design basis is as follows: Slight enhancements were made to maintain the basic outline shape; Appropriately enhance and improve boundary fit. This significantly enhances the image, forcing the contour to closely match the foreground. The compensation intensity coefficient increases non-linearly (2→5→10) rather than in an arithmetic progression, aiming to produce a significant compensation effect at high speeds, allowing the weights to increase rapidly to cope with drastic scene changes. The weight coefficients of the foreground confidence constraint are compensated in reverse; the faster the descent rate, the greater the compensation strength, thus enhancing the foreground constraint in advance when the scene changes rapidly and preventing contour divergence. Gradient descent is used to iteratively evolve the Snake contour, with 30 iterations. In each iteration, the internal energy, image gradient energy, and foreground confidence constraint energy are calculated based on the current contour position, and then weighted and summed according to the weight coefficients determined in step 2 to update the position of each vertex. After the iteration converges, the contour curve of the current frame is output.

[0025] In one embodiment of the present invention, the improved Snake contour filling module, after filling the contour interior and generating a complete moving target image, further includes a contour post-processing filtering step: converting the converged contour curve to a polar coordinate system, arranging the polar radii in ascending order of polar angles with the geometric center of the contour as the pole; performing a discrete cosine transform on the polar radii sequence, retaining the first 12.5% ​​of the low-frequency coefficients and then performing an inverse discrete cosine transform; combining the filtered polar radii sequence with the original polar angles, converting back to the vertex coordinates in the Cartesian coordinate system, and connecting each vertex to output the accurate target contour.

[0026] Specifically, the contour curve output by the improved Snake contour filling module after iterative convergence is obtained. This curve consists of a series of vertex coordinates connected sequentially in a clockwise direction, and the number of vertices is denoted as . (In this embodiment) (Use an integer between 60 and 120, depending on the complexity of the target contour); calculate the geometric center of the contour, which is the average of the x-coordinates and y-coordinates of all vertices. Take this geometric center as the pole. Traverse each vertex on the contour and calculate the polar radius (Euclidean distance from the vertex to the pole) and polar angle (the angle between the line connecting the vertex and the pole relative to the horizontal to the right) of that vertex. Sort all vertices in ascending order of polar angle, obtaining a polar radius sequence that corresponds one-to-one with the polar angle sequence. ,in, Indicates the first The polar radius values ​​corresponding to each polar angle; for the polar radius sequence Performing the Discrete Cosine Transform (DCT) transforms the polar radius sequence from the spatial domain to the frequency domain. The transformation formula is as follows: ,in, The length of the polar radius sequence (equal to the number of contour vertices). For the sorted number One extreme diameter value, For frequency index, Corresponding DC component, The larger the value, the higher the frequency. For the transform coefficients of the i-th frequency component, the discrete cosine transform has good energy compression characteristics: the main shape information of the contour is concentrated in the low-frequency coefficients ( The smaller components), while details such as jagged noise and local jitter on the contour are distributed in the high-frequency coefficients ( (Components with larger values); retain the first 12.5% ​​of the low-frequency coefficients after transformation, i.e., retain the index. The coefficients were then processed, and all remaining high-frequency coefficients were set to zero. The retention ratio of 12.5% ​​was chosen because a 1 / 8 retention ratio can retain the main shape features of the contour while filtering out high-frequency noise. Experiments have verified that this ratio achieves a good balance between smoothing effect and detail preservation. An inverse discrete cosine transform was performed on the coefficient sequence after retaining the low-frequency coefficients (high-frequency coefficients were set to zero) to reconstruct the filtered polar radius sequence. The formula for the inverse discrete cosine transform is: ,in, The transform coefficients are the high-frequency coefficients after being set to zero; the filtered polar radius sequence is... Combine the original polar angle sequence obtained in step 2 with the corresponding positions, and use the following formula to convert it back to the vertex coordinates in the Cartesian coordinate system: , ,in, For the filtered first Cartesian coordinates of the vertices, The coordinates of the geometric center calculated in step 2. The filtered first One extreme diameter value, For the first The original polar angles corresponding to each vertex are used to construct the target profile. All the restored vertices are connected sequentially in their original order to form a closed contour curve, which is output as the precise target profile. This profile has been filtered to remove high-frequency jagged noise while retaining the main shape features of the target. All pixels inside the precise target profile are assigned white (pixel value 255), and pixels outside the profile and on the boundary are assigned black (pixel value 0), generating a complete binary image of the moving target for use by the subsequent dual-branch mutual attention behavior discrimination module.

[0027] In one embodiment of the present invention, in the dual-branch mutual attention behavior discrimination module, the first branch calculates the temporal variation features of the target centroid displacement and the aspect ratio of the rectangle and encodes the features into an attention prior vector, including: Extract the contour of the moving target from the stable foreground mask, calculate the geometric center of the contour as the centroid coordinates, and calculate the minimum bounding rectangle of the contour to obtain the width and height. The centroid coordinate sequence and width and height sequence are continuously acquired within L frames. Each frame is used as a graph node, and the centroid displacement and aspect ratio change rate between adjacent frames are used as edge features between nodes to construct a spatiotemporal graph. A graph convolutional network is used to extract features from the spatiotemporal graph. The centroid coordinates and aspect ratio of each frame are used as the initial feature vectors of the nodes, and the centroid displacement and aspect ratio change rate between adjacent frames are used as edge weights. The node information of adjacent frames and the node information of one frame apart are aggregated through three graph convolutional layers to generate the updated features of each node. Global average pooling is performed on the updated features of each node to obtain a graph-level feature vector, which is then output as the attention prior vector.

[0028] Specifically, the contour of the moving target is extracted from the aforementioned stable foreground mask, and the geometric center of the contour is calculated as the centroid coordinates. ,in, This is the mean of the x-coordinates of all pixels within the contour. Using the mean of the ordinate, calculate the minimum bounding rectangle of the contour to obtain the width of the rectangle. and height Aspect Ratio Continuous data acquisition Centroid coordinate sequence of each frame within a frame and aspect ratio sequence In this embodiment Taking 15 frames, with a temporal window of approximately 0.6 seconds, calculate the centroid displacement between adjacent frames: That is, the Euclidean distance between the centroids of two frames, to obtain the centroid displacement sequence. Calculate the aspect ratio change rate between adjacent frames. The aspect ratio change rate sequence was obtained. Each frame is a graph node, totaling... A spatiotemporal graph is constructed using nodes and the centroid displacement and aspect ratio change rate between adjacent frames as edge features between nodes. ,in, For a set of nodes, The graph convolutional network uses a three-layer graph convolutional layer structure, with the centroid coordinates and aspect ratio of each frame as the set of edges. splicing as nodes initial feature vector With a dimension of 3, the centroids of adjacent frames are shifted. and aspect ratio change rate As an edge The weight of the reverse edge. The weights are the same, and nodes separated by one frame (e.g.) and (No edge connections, information is only passed between adjacent nodes) The propagation rule of the graph convolutional layer is as follows: ,in, For the first Layer nodes eigenvectors, It is the ReLU activation function. For nodes The set of neighboring nodes (containing nodes) (Itself and adjacent frame nodes). For nodes The degree, i.e., the neighboring nodes The number of elements in the middle, For the first The learnable weight matrix of the layer, For the first Layer nodes eigenvectors, For the first The bias term vector of each layer; through the first graph convolutional layer, each node aggregates information from itself and neighboring frame nodes, outputting a feature vector of dimension 64; through the second graph convolutional layer, each node further aggregates information from nodes one frame apart, outputting a feature vector of dimension 128; through the third graph convolutional layer, the output feature vector of dimension 256 is used as the update feature of each node; global average pooling is performed on the update features of each node: all The 256-dimensional feature vectors of each node are summed according to their dimensions and then divided by . A 256-dimensional graph-level feature vector is obtained, and the graph-level feature vector is output as the attention prior vector to drive the adaptive template matching of the second branch. When the displacement value of any frame in the centroid displacement sequence exceeds the first preset threshold (10 pixels in this embodiment), or the change rate of any frame in the aspect ratio change rate sequence exceeds the second preset threshold (0.3 in this embodiment), the current frame is determined to be an abnormal behavior candidate, and the second branch is triggered to perform template matching. The training process of the graph convolutional network is as follows: Normal behavior video clips from the publicly available behavior recognition dataset CAVIAR, including behaviors such as walking, standing, and sitting, are used. A total of 2000 sample clips are selected, each clip being 15 frames long. Simultaneously, negative sample pairs are generated by randomly permuting the frame order in the normal behavior sequence for contrastive learning. The weight matrices of the three graph convolutional layers are initialized using a Xavier uniform distribution, and the bias terms are initialized to zero vectors. The contrastive loss function is used, and its calculation formula is as follows: ,in, For batch size, The sample labels are 1 for positive sample pairs and 0 for negative sample pairs. Euclidean distance. For the graph-level feature vector of the anchor point sample, For positive samples, the graph-level feature vectors are... The graph-level feature vectors of negative samples. The distance interval parameter (set to 1 in this embodiment) is used to determine the objective of the loss function: the distance between positive sample pairs should approach 0, and the distance between negative sample pairs should be greater than 1. The Adam optimizer is used with an initial learning rate of 0.001, which decays to 0.5 times the previous learning rate every 50 rounds. The training rounds are 200. Training is terminated early when the loss value fluctuates less than 0.001 for 10 consecutive rounds. After training, the graph convolutional network parameters are fixed for subsequent inference stages.

[0029] In one embodiment of the present invention, the second branch receives the attention prior vector and performs adaptive template matching, including: Each standard behavior template in the template library contains a template feature vector and its corresponding binary image of the moving target. The template feature vector is generated by encoding the historical behavior sequence corresponding to the template through the same graph convolutional network and is in the same latent space as the attention prior vector. The attention prior vector is used to calculate the Euclidean distance with the template feature vector of each standard behavior template, and the top 3 templates with the smallest Euclidean distance are selected as the candidate template set. An affine transformation matrix is ​​constructed based on the centroid displacement component in the attention prior vector, and a rotation angle of is applied to the binary image of the moving target in the candidate template. The rotation transformation and scale factor are The scaling transformation, where, It is equal to the value of the angle of displacement of the center of mass. It is equal to the ratio of the effective bounding box area of ​​the target in the current frame to the bounding box area in the previous frame; Anisotropic scaling is performed on the candidate template based on the aspect ratio change rate component in the attention prior vector. The intersection-union ratio (IUR) of the aligned candidate template and the moving target image of the current frame is calculated pixel by pixel, and the maximum IUR is taken as the matching confidence output.

[0030] Specifically, each standard behavior template in the template library contains two parts: a template feature vector and its corresponding binary image of the moving target. The template feature vector is generated by encoding the historical behavior sequence corresponding to the template using a graph convolutional network identical to the first branch. The network structure, number of layers, activation function, and training parameters of this graph convolutional network are consistent with those of the first branch. Therefore, the template feature vector and the attention prior vector output by the first branch are in the same latent space. The same latent space means that the feature distributions of the two are aligned, and the Euclidean distance can be directly used to measure the matching confidence. The Euclidean distance is calculated between the attention prior vector and the template feature vector of each standard behavior template in the template library. The formula for calculating the Euclidean distance is as follows: ,in, The first of the attention prior vectors dimensional components, The first feature vector of the template dimensional components, The total dimension of the feature vector is 256 (in this embodiment, it is 256). The smaller the distance, the more similar the current behavior is to the template. The templates are sorted by Euclidean distance from smallest to largest, and the top M templates with the smallest distance are selected as the candidate template set. In this embodiment, M is 3. The centroid displacement component is extracted from the attention prior vector. This component includes two parameters: displacement direction angle and displacement amplitude. Let the centroid displacement direction angle be... The displacement amplitude is (Unit: pixels), construct the affine transformation matrix as follows: Among them, rotation angle equal to the direction angle of the center of mass displacement , scale factor The scale factor is equal to the ratio of the effective bounding box area of ​​the target in the current frame to the bounding box area of ​​the target in the previous frame. Directly using the linear derivation of the scale factor based on the displacement amplitude will lead to perspective distortion. The system uses the area ratio and combines it with the prior parameters of depth estimation for normalization calculation to eliminate the drastic scale changes caused by changes in camera viewpoint, so that the template can better adapt to the size changes of the target. The above affine transformation matrix is ​​applied to the binary image of the moving target in the candidate template, and the following operations are performed: Rotation transformation: rotate the image around its geometric center. Angle: aligns the template's pose with the current moving target; and scaling transformation: synchronously scales the image in both width and height directions. This multiplies the size of the template to match the current moving target.

[0031] In one embodiment of the present invention, the anisotropic scaling adjustment of the candidate template includes: Let the aspect ratio change rate component be... The scaling factor in the width direction is The scaling factor in the height direction is ; and satisfy: and Solving for the given information yields the following results. and ; Multiply the original width of the candidate template by After obtaining the adjusted width, multiply the original height of the candidate template by... After obtaining the adjusted height, perform independent resampling in the width and height directions on the binary image of the moving target in the candidate template to generate anisotropically scaled templates.

[0032] Specifically, the aspect ratio change rate component is extracted from the attention prior vector, denoted as... This component, calculated from the first branch, reflects the degree of aspect ratio change of the moving target between consecutive frames, and its value range is... ;when When the aspect ratio remains unchanged; when "When" indicates that the target has become taller and thinner (e.g., a person changes from standing to raising their hand); when This indicates that the target has become shorter and fatter (e.g., a person changes from standing to squatting). To ensure that anisotropic scaling accurately reflects changes in the target shape while avoiding excessive template deformation, the following two constraint equations are established: When the target becomes tall and thin, the width is reduced to half the height, making the template taller and thinner; when the target becomes short and wide, the width is increased to twice the height, making the template short and wide. This ensures that the area of ​​the template remains unchanged before and after scaling, preventing the template from being overstretched and distorted, or overcompressed and losing details, and adapting only to the changes in the target in terms of shape. Let the original width of the candidate template be... The original height is Calculate the adjusted width Adjusted height Independent resampling in the width and height directions is performed on the binary image of the moving target in the candidate template: Width direction: Bilinear interpolation is used to reduce the image width from... Zoom to In the height direction: bilinear interpolation is used to adjust the image height from... Zoom to After independent resampling, an anisotropically scaled template image is generated. While keeping the area unchanged, the shape of the template has been adaptively adjusted according to the aspect ratio change rate of the target, which can better match the actual shape of the moving target in the current frame.

[0033] In one embodiment of the present invention, the step of feeding back the matching confidence to the first branch to correct its threshold includes: Obtain the matching confidence of the second branch output. ,Establish Compared with the original threshold of the first branch Dynamic correction function between ,in, To preset the maximum correction step size, Sensitivity index; When matching confidence Three consecutive frames are greater than the lock threshold When the threshold correction operation is paused, the current threshold is maintained unchanged until two consecutive frames are passed. ≤ Then, restore the threshold correction; This serves as a new threshold for determining the first branch as an abnormal candidate.

[0034] Specifically, The closer the value is to 1, the higher the degree of matching between the template and the current moving target, indicating that the current behavior is closer to normal behavior; The closer the value is to 0, the lower the matching degree, indicating that the current behavior is more likely to be abnormal. Let the original anomaly detection threshold for the first branch be (calculated by acquiring a normal behavior video segment, calculating the centroid displacement and aspect ratio change rate of each frame, and taking three times the standard deviation of their mean as the threshold; in this embodiment, the calibration result is 0.5). This threshold is used to determine whether the centroid displacement or the rate of change of aspect ratio exceeds the normal range; among which, To preset the maximum correction step size, based on The 30% setting, that is This ratio ensures that the magnitude of a single correction does not exceed 30% of the original threshold, thus avoiding over-correction. The sensitivity index controls the correction amount. The sensitivity is set to 2 in this embodiment to obtain the best convergence effect; To prevent repeated fluctuations in the threshold under stable conditions, a locking mechanism is implemented: the matching confidence of the most recent 3 frames is continuously calculated. Value, if 3 consecutive frames Average value > Lock threshold When this occurs, it indicates that the system is in a high-confidence stable state, so the threshold correction operation is paused, and the current threshold is maintained unchanged; continuous monitoring continues. Value, when two consecutive frames All less than or equal to When the lock is released, the threshold correction operation resumes; where the lock threshold is... By acquiring a high-confidence video of normal behavior (target pose complete, unobstructed, and smooth movement), the matching confidence of each frame is calculated. Value, take all The mean of the values ​​minus one standard deviation is used as the locking threshold; the corrected values ​​are then... As a new threshold for the first branch anomaly candidate determination, it is used to compare and determine the centroid displacement and aspect ratio change rate in subsequent frames. When the centroid displacement or aspect ratio change rate exceeds... When an abnormal behavior candidate is identified, the second branch is triggered to perform template matching; the above steps are repeated to achieve online dynamic adjustment of the threshold.

[0035] In one embodiment of the present invention, the timing memory verification module includes: Maintain a circular memory pool of length N frames. Store the frame-level judgment result of each frame output by the dual-branch mutual attention behavior discrimination module into the circular memory pool. The frame-level judgment result includes an abnormal frame identifier and a matching confidence level. ; Count the number of abnormal frames marked as true in the circular memory pool, calculate the proportion of abnormal frames, and output an early warning signal when the proportion of abnormal frames is greater than the dynamic threshold; The dynamic threshold is calculated by extracting the matching confidence scores corresponding to normal frames in the circular memory pool. Sequence, calculation mean of the sequence with standard deviation ,set up: ,in, For dynamic thresholds, This is the sensitivity adjustment coefficient; After an early warning signal is output, a segment with more than K consecutive normal frames is extracted from the circular memory pool. The moving target images of each frame in the segment are weighted and fused to generate a new template. It is then determined whether the current template library capacity has reached its limit: if it has not reached the limit, it is directly expanded into the library; if it has reached the limit, the old template with the lowest usage frequency in the template library is replaced with the new template; when multiple old templates have the same lowest usage frequency, the old template with the earliest entry time is replaced according to the first-in-first-out principle.

[0036] Specifically, the abnormal frame identifier (1 for abnormal, 0 for normal) is extracted from the frame-level judgment result. The memory pool adopts a circular structure, with new data overwriting the oldest data, and always maintaining the record of the most recent N frames. The percentage of abnormal frames is calculated as follows: ,in, The number of abnormal frames. This is the total length of the memory pool. The percentage of abnormal frames; The specific method for calculating the dynamic threshold is as follows: Normal frames with an abnormal frame flag of 0 are selected from the memory pool, and the corresponding values ​​of these normal frames are extracted. Value, composition Sequence, calculation The mean and standard deviation of the sequence are used to derive the dynamic threshold. The mean represents the average level of confidence in matching normal behavior, while the standard deviation reflects the stability of normal behavior. , The sensitivity adjustment coefficient was determined by acquiring three different types of surveillance video segments: a high-stability scene (fixed lighting, target moving at a constant speed), a medium-stability scene (slight changes in lighting, target moving at varying speeds), and a low-stability scene (large fluctuations in lighting, target moving rapidly). The coefficient was iterated within the range of 0.1 to 1.0 with a step size of 0.1. For each video segment, calculate the false positive rate and false negative rate, and select the value that minimizes the sum of the average false positive rate and average false negative rate of the three video segments. In this embodiment, the calibration result is 0.5, a value that achieves a balance between high warning accuracy and low false alarm rate in various scenarios; when When an early warning signal is output, it triggers the early warning and storage linkage module. After the early warning signal is output, it extracts a segment with more than K consecutive normal frames from the circular memory pool (K=10 in this embodiment). The moving target images of each frame in the segment are weighted and fused: the frame closer to the current time has a higher weight, and the weight decreases linearly in time order. The new template generated by the weighted fusion replaces the old template with the lowest usage frequency in the template library. The usage frequency refers to the cumulative number of times each template is selected as a candidate template in the second branch (the initial value of the usage frequency of each template is 0. Each time the second branch performs adaptive template matching, the usage frequency counters of the M templates selected into the "candidate template set" (M=3 in this embodiment) are incremented by 1. If the same template is selected multiple times in the matching process of the same frame (for example, it is not selected repeatedly in the candidate template set), it is only incremented by 1). This mechanism ensures that the template library retains effective templates that are frequently matched and gradually eliminates old templates with low usage.

[0037] In one embodiment of the present invention, the early warning and storage linkage module includes: Maintain a circular video buffer to continuously store the original video frames and the corresponding moving target images; When the timing memory verification module outputs an early warning signal, it locates the continuous segment in the circular memory pool where the abnormal frame marker is true, and matches the confidence level within the continuous segment. The sequence is subjected to a one-dimensional moving average filter. The inflection points where the first derivative of the filtered sequence crosses zero and the second derivative is positive are calculated. If multiple inflection points exist, they are extracted. The global minimum inflection point with the smallest value and less than the preset absolute abnormal lower limit is used as the frame corresponding to the global minimum inflection point as the abnormal peak frame. Centered on the abnormal peak frame, based on the length of the abnormal continuous segment and minimum matching confidence Dynamically determine the truncation range: forward truncation length Backward truncation length ,in, and Preset positive coefficients; After the captured video clips are encoded and compressed, anomaly detection information is added to generate an alarm data packet, which is then uploaded to the cloud management platform. At the same time, the local alarm is triggered, and the alarm event is written to the local log database.

[0038] Specifically, a circular video buffer with a capacity of S seconds (S=30 in this embodiment) is constructed to continuously store the original video frames and their corresponding moving target images. The buffer adopts a circular overwrite mechanism, where new data automatically overwrites the oldest data to ensure that the video content of the most recent 30 seconds is always preserved. When the time sequence memory verification module outputs an early warning signal, the continuous segment in the circular memory pool where the abnormal frames are marked as true (the smallest interval in which multiple consecutive frames are marked as abnormal) is located, and the matching confidence of each frame within this continuous segment is obtained. ,constitute The sequence is subjected to a one-dimensional moving average filter; calculation First derivative of the sequence (adjacent frames) Find the position where the first derivative is zero and the second derivative is positive, based on the difference between the first and second derivatives. The inflection point where the sequence changes from decreasing to increasing is taken as the abnormal peak frame. This frame is the moment with the lowest matching confidence and is also the typical moment when abnormal behavior is most likely to occur. The preset absolute anomaly lower limit is obtained by the quantile method and the box plot method. Centered on the abnormal peak frame, the truncation range is dynamically determined based on the following two parameters: the length of the abnormal continuous segment. (Length of consecutive segments of anomalous frames (in frames), reflecting the duration of anomalous behavior) and minimum matching confidence. (the smallest within the abnormal continuous segment) The value reflects the severity of the abnormal behavior. The smaller the value, the more abnormal it is. and In this embodiment, a positive coefficient is preset. The value is 1.5. The value is 3.0. The longer the anomaly lasts, the more forward truncation occurs; the more severe the anomaly ( The smaller the size, the more backward cropping occurs; extracting from the circular video buffer. Instant The video clips, each a second long, are compressed using the H.265 encoding standard to reduce data size while maintaining image quality. Anomaly detection information, including the timestamp of the abnormal peak frame, the percentage of abnormal frames, and the minimum matching confidence, is added to the compressed video clips. These are then encapsulated into an alarm data packet, which is uploaded to the cloud management platform via the MQTT protocol. Simultaneously, the local audible and visual alarm is triggered, and the alarm event (timestamp, anomaly type, and video storage path, etc.) is written to the local log database.

[0039] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A video surveillance abnormal behavior real-time identification and security early warning system based on deep learning, characterized in that, Includes the following modules: The Gaussian Mixture and Frame Difference Selective Update Module is used to model the background of video frames and extract the foreground region by fusing the frame difference method. It selectively updates the background model based on motion history information to output a stable foreground mask. Improved Snake contour filling module: The improved Snake contour filling module is used to extract the target contour based on the stable foreground mask, extract the linear fitting slope of the structural similarity index of consecutive frames in real time, define the descent rate interval according to the slope, and perform reverse weighted compensation on the foreground confidence constraint term according to the corresponding compensation intensity coefficient matched in the interval, and then fill the interior of the converged contour to generate a complete moving target image. Dual-branch mutual attention behavior discrimination module: The first branch is used to calculate the temporal change features of the target centroid displacement and the aspect ratio of the rectangle. When the value is greater than the threshold, it is judged as an abnormal candidate, and the features are encoded into attention prior vectors. The second branch is used to receive the attention prior vector, perform adaptive template matching, output the matching confidence of the current frame, and feed the matching confidence back to the first branch. The first branch uses the matching confidence of the current frame to correct its dynamic threshold for geometric feature determination, and applies the corrected threshold to the anomaly candidate determination of the next frame; at the same time, if the matching confidence of the current frame is less than or equal to the preset anomaly confirmation threshold, the current frame is determined to be an anomaly. Timing memory verification module: used to maintain the memory pool. When the proportion of abnormal frames is greater than the dynamic threshold, it outputs an early warning signal and uses the normal templates in the memory pool to update the template library of the second branch online. Early warning and storage linkage module: used to trigger an alarm and upload abnormal video clips to the cloud when an early warning signal is output.

2. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, In the Gaussian mixture and frame difference selective update module, the background model is selectively updated based on motion history information, including: A motion history linked list is established for each pixel to record the foreground determination state sequence of the pixel in past consecutive video frames; When there is a continuous segment of foreground states in the state sequence, the length of the continuous segment does not exceed 3 frames, and the frame before and the frame after the continuous segment are both background states, it is determined that the pixel is in a hole region inside the moving target, the Gaussian mixture model parameters corresponding to the pixel are rolled back to the historical parameter values ​​before the start of the continuous segment, and a forced background filling operation is performed on the pixel. When the length of the continuous segment of the foreground state in the state sequence increases frame by frame, the pixel is marked as the stable motion core area, and the background model parameter update of its area is paused. The foreground mask after rollback filling and pause update processing is output as a stable foreground mask.

3. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, In the improved Snake contour filling module, the target contour is extracted based on the stable foreground mask, including: Calculate the structural similarity index between the current frame and the current background model. The weighting coefficients of the prospect confidence constraint term. Set as: The weighting coefficient of the internal energy term is set to... ,in, The preset maximum weighting coefficient; Using the converged contour vertex coordinates of the previous frame as the initial position of the Snake model in the current frame, the normal direction displacement of each vertex of the contour in the previous frame is calculated, and a two-dimensional all-zero matrix with the same resolution as the current frame image is constructed. The normal direction displacement of each vertex is used as a pixel value and mapped to the corresponding vertex coordinate position in the two-dimensional all-zero matrix to generate a two-dimensional sparse displacement field matrix. The two-dimensional sparse displacement field matrix is ​​convolved with the gradient field of the two-dimensional image of the current frame by a size of 3*3 pixels to generate an initial velocity field that guides the evolution of the contour in the current frame. For 5 consecutive frames For video segments where the value decreases frame by frame, calculate the value within 5 frames. The slope of the linear fit of the value is used as the descent rate; the descent rate is divided into three intervals: low-speed, medium-speed, and high-speed, each corresponding to a different compensation intensity coefficient. The appropriate compensation intensity coefficient is selected based on the interval in which the descent rate falls. Perform reverse compensation, whereby, The weighting coefficients after compensation. The weighting coefficients before compensation. To compensate for the strength coefficient, The rate of decrease.

4. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, The improved Snake contour filling module, after filling the contour and generating a complete moving target image, also includes a contour post-processing filtering step: converting the converged contour curve to a polar coordinate system, arranging the polar radii in ascending order of polar angles with the geometric center of the contour as the pole; performing a discrete cosine transform on the polar radii sequence, retaining the first 12.5% ​​of the low-frequency coefficients, and then performing an inverse discrete cosine transform; combining the filtered polar radii sequence with the original polar angles, converting back to the vertex coordinates in the Cartesian coordinate system, and connecting each vertex to output the accurate target contour.

5. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, In the dual-branch mutual attention behavior discrimination module, the first branch calculates the temporal variation features of the target centroid displacement and the aspect ratio of the rectangle and encodes these features into an attention prior vector, including: Extract the contour of the moving target from the stable foreground mask, calculate the geometric center of the contour as the centroid coordinates, and calculate the minimum bounding rectangle of the contour to obtain the width and height. The centroid coordinate sequence and width and height sequence are continuously acquired within L frames. Each frame is used as a graph node, and the centroid displacement and aspect ratio change rate between adjacent frames are used as edge features between nodes to construct a spatiotemporal graph. A graph convolutional network is used to extract features from the spatiotemporal graph. The centroid coordinates and aspect ratio of each frame are used as the initial feature vectors of the nodes, and the centroid displacement and aspect ratio change rate between adjacent frames are used as edge weights. The node information of adjacent frames and the node information of one frame apart are aggregated through three graph convolutional layers to generate the updated features of each node. Global average pooling is performed on the updated features of each node to obtain a graph-level feature vector, which is then output as the attention prior vector.

6. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, The second branch receives the attention prior vector and performs adaptive template matching, including: Each standard behavior template in the template library contains a template feature vector and its corresponding binary image of the moving target. The template feature vector is generated by encoding the historical behavior sequence corresponding to the template through the same graph convolutional network and is in the same latent space as the attention prior vector. The attention prior vector is used to calculate the Euclidean distance with the template feature vector of each standard behavior template, and the top 3 templates with the smallest Euclidean distance are selected as the candidate template set. An affine transformation matrix is ​​constructed based on the centroid displacement component in the attention prior vector, and a rotation angle of is applied to the binary image of the moving target in the candidate template. The rotation transformation and scale factor are The scaling transformation, where, It is equal to the value of the angle of displacement of the center of mass. It is equal to the ratio of the effective bounding box area of ​​the target in the current frame to the bounding box area in the previous frame; Anisotropic scaling is performed on the candidate template based on the aspect ratio change rate component in the attention prior vector. The intersection-union ratio (IUR) of the aligned candidate template and the moving target image of the current frame is calculated pixel by pixel, and the maximum IUR is taken as the matching confidence output.

7. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 6, characterized in that, The anisotropic scaling adjustment of the candidate template includes: Let the aspect ratio change rate component be... The scaling factor in the width direction is The scaling factor in the height direction is ; and satisfy: and Solving for the given information yields the following results. and ; Multiply the original width of the candidate template by After obtaining the adjusted width, multiply the original height of the candidate template by... After obtaining the adjusted height, perform independent resampling in the width and height directions on the binary image of the moving target in the candidate template to generate anisotropically scaled templates.

8. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, The step of feeding back the matching confidence score to the first branch to correct its threshold includes: Obtain the matching confidence of the second branch output. ,Establish Compared with the original threshold of the first branch Dynamic correction function between ,in, To preset the maximum correction step size, Sensitivity index; When matching confidence Three consecutive frames are greater than the lock threshold When the threshold correction operation is paused, the current threshold is maintained unchanged until two consecutive frames are passed. ≤ Then, restore the threshold correction; This serves as a new threshold for determining the first branch as an abnormal candidate.

9. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, The time-series memory verification module includes: Maintain a circular memory pool of length N frames. Store the frame-level judgment result of each frame output by the dual-branch mutual attention behavior discrimination module into the circular memory pool. The frame-level judgment result includes an abnormal frame identifier and a matching confidence level. ; Count the number of abnormal frames marked as true in the circular memory pool, calculate the proportion of abnormal frames, and output an early warning signal when the proportion of abnormal frames is greater than the dynamic threshold; The dynamic threshold is calculated by extracting the matching confidence scores corresponding to normal frames in the circular memory pool. Sequence, calculation mean of the sequence with standard deviation ,set up: ,in, For dynamic thresholds, This is the sensitivity adjustment coefficient; After an early warning signal is output, a segment with more than K consecutive normal frames is extracted from the circular memory pool. The moving target images of each frame in the segment are weighted and fused to generate a new template. It is then determined whether the current template library capacity has reached its limit: if it has not reached the limit, it is directly expanded into the library; if it has reached the limit, the old template with the lowest usage frequency in the template library is replaced with the new template; when multiple old templates have the same lowest usage frequency, the old template with the earliest entry time is replaced according to the first-in-first-out principle.

10. The video surveillance abnormal behavior real-time identification and security early warning system based on deep learning according to claim 1, characterized in that, The early warning and storage linkage module includes: Maintain a circular video buffer to continuously store the original video frames and the corresponding moving target images; When the sequential memory verification module outputs an early warning signal, it locates the continuous segment in the circular memory pool where the abnormal frame is marked as true, and matches the confidence level within the continuous segment. The sequence is subjected to a one-dimensional moving average filter. The inflection points where the first derivative of the filtered sequence crosses zero and the second derivative is positive are calculated. If multiple inflection points exist, they are extracted. The global minimum inflection point with the smallest value and less than the preset absolute abnormal lower limit is used as the frame corresponding to the global minimum inflection point as the abnormal peak frame. Centered on the abnormal peak frame, based on the length of the abnormal continuous segment and minimum matching confidence Dynamically determine the truncation range: forward truncation length Backward truncation length ,in, and Preset positive coefficients; After the captured video clips are encoded and compressed, anomaly detection information is added to generate an alarm data packet, which is then uploaded to the cloud management platform. At the same time, the local alarm is triggered, and the alarm event is written to the local log database.

Images