Intelligent security video monitoring system for preventing animal aggression

Abstract

The application discloses an intelligent security video monitoring system for preventing animal attack behaviors, and relates to the technical field of intelligent security, which comprises the following steps: synchronously collecting a thermal radiation image sequence and an RGB image sequence to generate a cross-modal video stream data packet; performing multispectral feature extraction on the cross-modal video stream data packet to generate visible light enhanced features and infrared specific features; extracting cross-modal shared features through a shared weight convolution layer from the visible light enhanced features and the infrared specific features, splicing to form a mixed feature tensor; horizontally dividing the mixed feature tensor into equal-width regions, calculating regional pseudo-anchor points, and generating multi-scale candidate boxes through an improved YOLOv7 detection head, guiding the same type aggregation and the different type repulsion loss through the pseudo-anchor points, and outputting a high confidence candidate box set; and converting the boundary box parameters of each candidate box in the high confidence candidate box set into a state vector. The application can ensure the alignment of cross-modal data timestamps by dynamically adjusting primary and secondary data streams.

Description

Technical Field

[0001] This invention relates to the field of intelligent security technology, and in particular to an intelligent security video surveillance system for preventing animal attacks. Background Technology

[0002] In recent years, intelligent security systems have gradually adopted multimodal sensing technology in scenarios such as wildlife protection and border monitoring, enhancing all-weather monitoring capabilities by integrating visible light and infrared imaging. Visible light cameras acquire RGB images through CMOS sensors and process them in real time using an ISP; infrared cameras rely on microbolometers to capture thermal radiation information. The two systems switch between independent data streams in a time-division manner to adapt to changes in lighting conditions. Multispectral feature extraction methods, such as visible light enhancement technology based on frequency domain phase spectrum and infrared high-frequency feature extraction using wavelet transform, are combined with deep learning models to achieve multi-scale target detection. Furthermore, dynamic threat assessment typically employs single-modal trajectory analysis or attitude estimation, such as a combination of OpenPose keypoint detection and LSTM temporal modeling.

[0003] However, existing technologies still have significant drawbacks: current systems rely on fixed thresholds or manually set illumination intensities to switch between primary and secondary data streams, leading to switching delays when the visible light signal-to-noise ratio drops sharply under low illumination. Furthermore, they fail to address cross-modal timestamp synchronization errors, affecting subsequent fusion accuracy. Traditional methods directly stitch infrared and visible light features together, ignoring the complementarity between the frequency domain phase spectrum and high-frequency textures, and lack a lightweight shared-weight convolutional architecture, resulting in computational redundancy and feature obfuscation. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an intelligent security video surveillance system for preventing animal attacks, which solves the problem of switching delay in existing intelligent security systems when the visible light signal-to-noise ratio drops sharply in low light conditions under dynamic environments.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides an intelligent security video surveillance system for preventing animal aggression, comprising a synchronous acquisition module that synchronously acquires thermal radiation image sequences and RGB image sequences to generate cross-modal video stream data packets;

[0008] The feature extraction module performs multispectral feature extraction on cross-modal video stream data packets to generate visible light enhanced features and infrared specific features;

[0009] The feature fusion module extracts cross-modal shared features from visible light enhancement features and infrared specific features through a shared weight convolutional layer, and then concatenates them to form a hybrid feature tensor.

[0010] The detection and tracking module divides the hybrid feature tensor into equal-width regions, calculates pseudo-anchor points in the regions, and generates multi-scale candidate boxes by improving the YOLOv7 detection head. The pseudo-anchor points guide the aggregation of similar categories and the rejection of dissimilar categories, and output a set of high-confidence candidate boxes.

[0011] The evaluation and analysis module transforms the bounding box parameters of each candidate box in the high-confidence candidate box set into a state vector, and generates an animal target list by combining Kalman filtering and cross-frame feature association.

[0012] The graded response module extracts movement trajectories from the list of animal targets, combines them with OpenPose keypoint detection, and integrates historical behavioral features into the LSTM-TCN hybrid model to output attack probabilities and map them to dynamic threat levels, triggering a graded response mechanism.

[0013] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression described in this invention, the synchronous acquisition of thermal radiation image sequences and RGB image sequences includes the following steps:

[0014] The ambient light intensity is detected by the light intensity sensor of the visible light camera. When the light intensity is higher than the average light intensity value, RGB image acquisition is initiated, and when the light intensity is lower than the average light intensity value, infrared thermal imaging acquisition is initiated.

[0015] The main control chip synchronously sends VSYNC vertical synchronization pulses, and aligns the timestamps of the thermal radiation image sequence and the RGB image sequence through the FPGA timer, compressing them into independent infrared video streams and visible light video streams;

[0016] Based on the signal-to-noise ratio detection results of the visible light video stream, the primary and secondary data streams are dynamically switched to generate cross-modal video stream data packets.

[0017] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression described in this invention, the multispectral feature extraction includes the following steps:

[0018] The three-channel matrix of the RGB image is separated, and a complex frequency domain matrix is ​​generated by two-dimensional fast Fourier transform and Hanning window function. The phase spectrum components of each channel are extracted and superimposed with the three-channel grayscale matrix to generate visible light enhancement features.

[0019] A three-level discrete wavelet transform is performed on the thermal radiation image, which is then decomposed into a third-level diagonal high-frequency subband using the Haar wavelet basis. After Sobel gradient calculation and non-maximum suppression, specific infrared features are generated.

[0020] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression described in this invention, the extraction of cross-modal shared features includes the following steps:

[0021] The three-channel data of the visible light enhancement feature is concatenated with the single-channel data of the infrared specific feature to form a four-channel input matrix;

[0022] By sequentially performing large convolutional kernel feature downsampling, max pooling compression, and small convolutional kernel feature enhancement through shared weight convolutional layers, cross-modal shared features are output.

[0023] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression described in this invention, the step of splicing together to form a hybrid feature tensor includes the following steps:

[0024] The spatial dimension of cross-modal shared features is uniformly divided into grid regions. Global average pooling is performed on each grid to generate local feature descriptors, which are then compressed into local feature matrices through fully connected layers.

[0025] Adjust the number of channels for infrared-specific features and visible light enhancement features to match the spatial resolution of the local feature matrix;

[0026] The local feature matrix, the adjusted infrared-specific features, and the visible light enhancement features are concatenated along the channel dimension to form a hybrid feature tensor.

[0027] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression described in this invention, the step of generating multi-scale candidate boxes by improving the YOLOv7 detection head includes the following steps.

[0028] The hybrid feature tensor is divided into non-overlapping regions of equal width, and the average value of the infrared and visible light feature vectors within the regions is extracted to generate a set of pseudo anchor points.

[0029] The set of pseudo-anchor points is input into the improved YOLOv7 detection head for multi-scale feature map convolution processing to generate multi-scale candidate boxes.

[0030] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression as described in this invention, the pseudo-anchor point guided homogeneous aggregation and heterogeneous exclusion loss refers to applying homogeneous aggregation constraints to the infrared specific features of the candidate box, driving homogeneous features to gather towards the pseudo-anchor point, and applying heterogeneous exclusion constraints to the left and right adjacent blocks of the block in which the candidate box is located, suppressing cross-block feature confusion.

[0031] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression described in this invention, the generation of the animal target list includes the following steps:

[0032] The bounding box parameters of each candidate box in the high-confidence candidate box set are transformed into a state vector, and associated with infrared pseudo-anchor features and visible light-enhanced pseudo-anchor features.

[0033] Predict the current target position based on the state vector, and update the state vector by matching Mahalanobis distance to generate a motion-corrected prediction box.

[0034] The corrected predicted bounding box is correlated across frames with infrared pseudo-anchor features and visible light enhanced pseudo-anchor features. The spatiotemporal-feature comprehensive similarity is calculated. Continuous unmatched targets are determined to have left the monitoring area and removed from the tracking list, generating an animal target list.

[0035] As a preferred embodiment of the intelligent security video surveillance system for preventing animal aggression as described in this invention, the step of outputting the attack probability and mapping it to a dynamic threat level includes the following steps:

[0036] Based on the angular deviation between the movement direction angle in the animal target list and the direction angle of the protected area boundary, the head height difference and limb opening degree detected by OpenPose, a multi-dimensional time-series vector is formed by combining information entropy;

[0037] The multidimensional time-series vector is input into the LSTM-TCN hybrid model, which outputs the attack probability and divides the threat levels into low, medium and high based on the proportion of points.

[0038] As a preferred embodiment of the intelligent security video surveillance system for preventing animal attacks described in this invention, the trigger-level response mechanism refers to: under low threat levels, using an AR rendering engine to overlay a dynamic tracking frame and pushing it to the monitoring terminal in real time; under medium threat levels, triggering an audible and visual alarm and sending alarm information to the SMS gateway; and under high threat levels, activating drone patrol and ultrasonic interference based on trajectory prediction.

[0039] The beneficial effects of this invention are as follows: Based on the signal-to-noise ratio detection of edge computing nodes, the primary and secondary data streams are dynamically adjusted. Combined with the VSYNC pulse synchronization and clock calibration of the FPGA timer, cross-modal data timestamp alignment errors are ensured, avoiding the problem of missed detection in low light caused by fixed threshold switching. The visible light frequency domain phase spectrum retains high-frequency texture details, and the infrared three-level wavelet decomposition extracts diagonal high-frequency subbands. Sparse features are generated through Sobel gradient magnitude and neighborhood non-maximum suppression. After bilinear interpolation alignment, a complementary feature matrix is ​​formed, improving the target discrimination in complex backgrounds. The hybrid feature tensor is horizontally divided into equal-width regions, and the mean values ​​of infrared and visible light features are calculated to generate a set of pseudo-anchor points. The improved YOLOv7 detection head drives similar target features to aggregate towards pseudo-anchor points through similar aggregation and dissimilar repulsion loss functions, suppressing cross-block interference and improving the detection accuracy of animal targets. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a system diagram of an intelligent security video surveillance system used to prevent animal attacks.

[0042] Figure 2 This is a schematic diagram of the synchronous acquisition module.

[0043] Figure 3 This is a schematic diagram of feature fusion and detection tracking.

[0044] Figure 4 This is a schematic diagram of a graded response mechanism. Detailed Implementation

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0046] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0047] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0048] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides an intelligent security video surveillance system for preventing animal aggression, comprising the following steps:

[0049] The synchronous acquisition module simultaneously acquires thermal radiation image sequences and RGB image sequences, generating cross-modal video stream data packets.

[0050] Specifically, the power interface of the infrared camera is connected to a DC power supply, and the power interface of the visible light camera is connected to a power supply of the same specification; the control bus of the infrared camera sends an initialization command, and the control bus of the visible light camera receives the initialization command to complete the device self-test and sensor preheating.

[0051] The CMOS sensor of the visible light camera outputs a brightness histogram in real time to calculate the average illuminance value of the scene. When the illuminance sensor of the visible light camera detects that the ambient light intensity reaches or exceeds the average illuminance value, the image signal processor (ISP) of the visible light camera starts Bayer format RGB image acquisition. When the illuminance sensor of the visible light camera detects that the ambient light intensity is lower than the average illuminance value, the uncooled microbolometer of the infrared camera starts the thermal imaging acquisition process, with the working band covering the mid- and far-infrared spectral range. The main control chip sends VSYNC vertical synchronization pulses to both the infrared and visible light cameras simultaneously. When the FPGA timers of the infrared and visible light cameras receive the rising VSYNC pulse, they acquire the thermal radiation image sequence and the RGB image sequence, record the local timestamp, and align the timestamps through clock calibration.

[0052] The timestamp-aligned thermal radiation image sequence is compressed into an infrared video stream, and the RGB image sequence is compressed into an independent visible light video stream using an encoder. The infrared and visible light video streams are respectively bound to edge computing nodes. The visible light video stream is decoded using a video decoder to extract the luminance component of each frame of the RGB image. Based on the histogram distribution of the luminance component, a signal-to-noise ratio (SNR) detection algorithm is used to calculate the SNR value of the current frame. The calculation formula is as follows:

[0053] ;

[0054] in, For signal-to-noise ratio, This represents the pixel mean of the image's luminance component. This represents the noise standard deviation of the image luminance component.

[0055] The logic controller of the edge computing node continuously receives the signal-to-noise ratio (SNR) calculation results of the visible light video stream. When the continuous SNR values ​​are all below the SNR threshold, the logic controller sends control commands to the infrared camera and the visible light camera, marking the thermal radiation image stream of the infrared camera as the main data stream and the RGB image stream of the visible light camera as the auxiliary data stream. Timestamp synchronization markers are added to the main data stream and the auxiliary data stream, and then encapsulated to generate cross-modal video stream data packets.

[0056] To further explain, the cross-modal data stream includes the device ID code, the current ambient light intensity measurement, and the primary and secondary data stream markers.

[0057] The signal-to-noise ratio (SNR) threshold is set based on the noise standard deviation of the human eye for the luminance component. Here, the standard deviation is set to 0.03, so the SNR is approximately 10.4. However, in practice, a safety margin needs to be reserved, and the specific setting should be made in combination with the quantization error of the edge computing nodes.

[0058] The feature extraction module performs multispectral feature extraction on cross-modal video stream data packets, generating visible light enhanced features and infrared specific features.

[0059] Specifically, the main data stream in the cross-modal video stream data packet is restored to a thermal radiation image sequence, and the auxiliary data stream is restored to an RGB image sequence; for the single-frame RGB image generated by the visible light video stream, the R channel matrix, G channel matrix and B channel matrix are separated; according to the standard luminance coefficient, the three-channel grayscale image is calculated pixel by pixel to generate a three-channel grayscale matrix of the same size as the original RGB image.

[0060] Spatially align and verify the three-channel grayscale matrix with the original RGB image. Perform two-dimensional Fast Fourier Transform (FFT) on the R, G, and B channels of the spatially aligned and verified original RGB image. Apply a Hanning window function to each color channel matrix to suppress spectral leakage. Obtain the complex frequency domain representation of each channel through FFT, and output the integrated complex frequency domain matrices of the R, G, and B channels. Extract the phase spectrum components from the complex frequency domain matrices and calculate the phase spectrum of each channel. Taking the R channel as an example, the expression is:

[0061] ;

[0062] in, Red Channel phase spectrum The matrix represents the frequency domain phase information of the red component in the original RGB image, used to preserve the high-frequency texture features of the image. Fourier transform result of the red channel The imaginary part of the number represents the orthogonal component (sine component) of the red component in the frequency domain, which, together with the real part, constitutes the complex frequency domain representation. Fourier transform result of the red channel The real part of represents the in-phase component (cosine component) of the red component in the frequency domain, and together with the imaginary part, it describes the frequency domain characteristics. The red channel is a complex frequency domain matrix after undergoing a two-dimensional Fast Fourier Transform (FFT), which converts the red pixel values ​​in the spatial domain into a frequency domain representation, including amplitude and phase information.

[0063] Repeat the same operation on the G and B channels to obtain and The three-channel phase spectrum is linearly mapped to generate a phase spectrum matrix of the same size as the three-channel grayscale matrix; the phase spectrum matrix and the three-channel grayscale matrix are superimposed at the pixel level, and the superposition result is truncated to generate visible light enhancement features.

[0064] The following steps describe how to perform a three-level discrete wavelet transform on a single frame of a thermal radiation image sequence:

[0065] The first-level decomposition uses the Haar wavelet basis to perform row and column filtering on a single frame of thermal radiation image to generate a first-level low-frequency sub-band, a horizontal high-frequency sub-band, a vertical high-frequency sub-band, and a diagonal high-frequency sub-band.

[0066] The second-level decomposition repeats row and column filtering operations on the low-frequency sub-bands obtained from the first-level decomposition to generate the second-level low-frequency sub-bands and their corresponding horizontal, vertical and diagonal high-frequency sub-bands.

[0067] The third-level decomposition further performs row and column filtering on the low-frequency subbands obtained from the second-level decomposition, and finally outputs the third-level low-frequency subbands and their corresponding horizontal, vertical and diagonal high-frequency subbands.

[0068] The diagonal high-frequency subband of the third-level decomposition is selected as the highest frequency component, and the significance of the coefficients is enhanced by absolute value operation. Sobel operator convolution is performed on the diagonal high-frequency subband after absolute value operation to extract the horizontal and vertical gradients and generate gradient magnitudes. Neighborhood comparison is performed on the gradient magnitudes along the gradient direction, and non-maximum suppression and binarization are performed to generate a sparse feature point matrix after binarization. The sparse feature point matrix after binarization is upsampled by bilinear interpolation algorithm to generate infrared specific features of the same size as the visible light enhancement features.

[0069] To further explain, the specific steps for performing neighborhood comparisons of gradient magnitudes along the gradient direction are as follows:

[0070] For the 0° direction, compare the magnitude of the current pixel with its left and right adjacent pixels; for the 45° direction, compare the magnitude of the current pixel with the magnitude of the pixels on the top left and bottom right diagonals; for the 90° direction, compare the magnitude of the current pixel with the magnitude of the pixels on the top and bottom; for the 135° direction, compare the magnitude of the current pixel with the magnitude of the pixels on the top right and bottom left diagonals, retain the local maxima points whose magnitudes are greater than those of the adjacent pixels, and set the rest to zero.

[0071] The feature fusion module extracts cross-modal shared features from visible light enhanced features and infrared specific features through a shared weight convolutional layer, and then concatenates them to form a hybrid feature tensor.

[0072] Specifically, visible light enhancement features and infrared-specific features are merged according to channel dimension. The three-channel data of visible light enhancement features and the single-channel data of infrared-specific features are concatenated to form a four-channel input matrix. An initial convolutional layer with shared weights is applied to reduce the dimensionality of the four-channel input matrix. The first layer uses a large convolutional kernel to downsample the features of the four-channel input matrix, reducing the spatial dimension and extracting basic features. The second layer further compresses the size of the basic feature map through max pooling while keeping the number of channels unchanged. The third layer uses a small convolutional kernel to adjust the channel dimension to achieve feature enhancement and output cross-modal shared features.

[0073] Check whether the spatial resolution of the cross-modal shared features is consistent with that of the infrared-specific features and the visible light-enhanced features. If there are dimensional differences, adjust them to the same size using bilinear interpolation to ensure that all feature maps are strictly aligned in the height and width dimensions.

[0074] The spatial dimension of the cross-modal shared features is uniformly divided into grid regions, with each grid covering pixels in the height and width directions. Global average pooling is performed on the cross-modal shared features within each grid region to compress them into local feature descriptors. A fully connected layer is applied to each local feature descriptor to compress the feature dimension and generate a local feature matrix.

[0075] Infrared-specific features and visible light-enhanced features are respectively adjusted for the number of channels using convolution kernels to maintain spatial resolution consistent with the local feature matrix; the local feature matrix, the adjusted infrared-specific features, and the visible light-enhanced features are then concatenated along the channel dimension to form a hybrid feature tensor.

[0076] The detection and tracking module divides the hybrid feature tensor into equal-width regions, calculates pseudo-anchor points in the regions, and generates multi-scale candidate boxes by improving the YOLOv7 detection head. The pseudo-anchor points guide the aggregation of similar categories and the rejection of dissimilar categories, and output a set of high-confidence candidate boxes.

[0077] Specifically, the hybrid feature tensor is uniformly divided into regions of equal width along the horizontal direction. Each region covers the entire vertical range and has an equal width, ensuring that the region divisions are non-overlapping and fully covered. Within each divided region, all feature vector sets corresponding to the spatial location are extracted from infrared specific features and visible light enhanced features, respectively. Based on the target location information in the detection results, feature vectors belonging to the animal category within each region are selected from all feature vector sets. The average value of the same type of target feature vectors in infrared specific features and visible light enhanced features is calculated, and used as the representative feature of the corresponding region. The horizontal center coordinates of the pseudo-anchor points are determined according to the region location, and the average values ​​of infrared and visible light features are mapped to the corresponding coordinate positions, respectively, generating a set of pseudo-anchor points with unique region numbers and modality type labels.

[0078] The set of pseudo-anchor points is input into the improved YOLOv7 detection head for multi-scale feature map convolution processing. The specific steps are as follows:

[0079] The first detection layer generates prediction results for large-sized targets through convolution, the second detection layer processes medium-sized targets through downsampling, and the third detection layer further processes small-sized targets through downsampling. Each detection layer outputs a prediction vector containing bounding box coordinates, confidence scores, and class probabilities. The coordinates are mapped to a standardized range, the confidence scores are converted into probability values, and the class probabilities are normalized through nonlinear transformations.

[0080] Prediction results at different scales are upsampled to a uniform resolution and then stitched together to form a multi-scale comprehensive prediction result. Finally, candidate boxes with a probability higher than the class probability threshold and a confidence level that meet the criteria are selected from the animal categories.

[0081] To further explain, the category probability threshold is used to filter the probability that a target belongs to the animal category. It represents the minimum confidence requirement for classifying a target as an animal. It is set according to the classification difficulty of the animal category and the data distribution. Generally, the value range is 0.3 to 0.6 (e.g., 0.5 is used for balanced multi-class detection, and 0.3 is used for small targets or occluded scenes).

[0082] The evaluation and analysis module transforms the bounding box parameters of each candidate box in the high-confidence candidate box set into a state vector, and generates an animal target list by combining Kalman filtering and cross-frame feature association.

[0083] Specifically, the center coordinates of the candidate boxes are scaled proportionally to the horizontal block index according to the original image width, and the effectiveness of separate numbering is verified. Based on the block number, infrared pseudo-anchor features and visible light enhanced pseudo-anchor features are extracted from the pseudo-anchor set. The dominant pseudo-anchor is selected according to the modal primary and secondary labels of the candidate boxes. When the main data stream is infrared, infrared pseudo-anchor features are selected, and when the auxiliary data stream is visible light, visible light enhanced pseudo-anchor features are selected. The same-class aggregation constraint is applied to the infrared specific features of the candidate boxes. The cosine similarity between the infrared specific features and the infrared pseudo-anchor features in the same block is calculated and minimized to drive the same-class features to cluster towards pseudo-anchors. The opposite-class exclusion constraint is applied to the left and right adjacent blocks of the block where the candidate box is located. The infrared pseudo-anchors of the adjacent blocks are retrieved. The cosine similarity between the infrared specific features and the adjacent pseudo-anchors is calculated, and the negative contribution of the cosine similarity is maximized to suppress cross-block feature confusion.

[0084] The loss function is constructed by combining the similarity calculation results of all candidate boxes. The gradient of the loss with respect to the weights of the improved YOLOv7 detection head is calculated by the chain rule. The Adam optimizer is used to update the convolution kernel parameters to enhance the consistency of features of similar targets and output a set of high-confidence candidate boxes.

[0085] The bounding box parameters (center coordinates, width and height) of each candidate box in the high-confidence candidate box set are transformed into a state vector. If the candidate box is marked as infrared dominant, it is associated with infrared pseudo-anchor features; if it is visible light dominant, it is associated with visible light enhanced pseudo-anchor features. The current target position is predicted based on the state vector, and the candidate boxes in the high-confidence candidate box set are used as observations. The state vector is updated by Mahalanobis distance matching to generate a predicted box with motion correction. The corrected predicted box is associated with infrared pseudo-anchor features and visible light enhanced pseudo-anchor features across frames. The spatiotemporal-feature comprehensive similarity is calculated by fusing the infrared pseudo-anchor features and visible light enhanced pseudo-anchor features. For targets that continuously fail to match the Kalman prediction, the tracking is terminated based on the similarity result. For targets that fail to match successfully for multiple consecutive frames, they are determined to have left the monitoring area and are removed from the tracking list, generating an animal target list.

[0086] To further explain, the animal target list includes the animal target ID, current frame bounding box coordinates, infrared pseudo-anchor features, visible light enhanced pseudo-anchor features, motion trajectory queue, and target visibility flag (0: normal, 1: predicted state).

[0087] The graded response module extracts movement trajectories from the list of animal targets, combines them with OpenPose keypoint detection, and integrates historical behavioral features into the LSTM-TCN hybrid model to output attack probabilities and map them to dynamic threat levels, triggering a graded response mechanism.

[0088] Specifically, the center coordinate data of the latest consecutive multiple frames are extracted from the trajectory queue of the animal target list, arranged in ascending order by timestamp, and the time interval is calculated based on the video stream frame rate; the horizontal and vertical displacements of two adjacent frames are calculated; and the instantaneous velocity is calculated by combining the horizontal and vertical displacements of two adjacent frames with the time interval, forming a velocity sequence, expressed as:

[0089] ;

[0090] in, For the goal at all times instantaneous speed, For horizontal displacement, For vertical displacement, For time intervals;

[0091] The instantaneous acceleration between two adjacent frames is calculated based on the instantaneous velocity, and an acceleration sequence corresponding to the velocity sequence is generated. Based on the velocity sequence and the acceleration sequence, the coordinates of the latest two frames are taken, and the changes in the horizontal and vertical directions are calculated to obtain the motion direction angle.

[0092] The boundary equation of the protected area is read from the preset parameter library; the expression is:

[0093] ;

[0094] in, To control the degree of inclination of the straight line in the horizontal direction, To control the degree of inclination of the straight line in the vertical direction, This is a constant term that determines the positional offset of the line. and These are the horizontal and vertical coordinate variables in a Cartesian coordinate system.

[0095] Based on the boundary line equation of the protected area, the boundary direction angle is calculated as follows:

[0096] ;

[0097] in, This represents the boundary direction angle.

[0098] If the boundary is a curve, then the tangent direction at the current position of the target is taken as the boundary direction angle.

[0099] Calculate the angular deviation between the motion direction angle and the boundary direction angle, and use the angular deviation as a threat assessment parameter for the target approaching the boundary.

[0100] Based on the calculated angular deviation between the motion direction angle and the boundary direction angle, and combined with the coordinates of the animal target detection box (center point, width, and height), pose analysis is performed as follows:

[0101] From the four-channel input matrix fused with visible light enhancement features and infrared specific features, a rectangular region is cropped, and a cropped multimodal image patch is output. This multimodal image patch is then input into a pre-trained OpenPose keypoint detection model, which generates multi-scale feature maps through a feature pyramid network, outputting heatmaps and affinity maps of the skeletal keypoints. Local maxima filtering is performed on the heatmap to suppress non-maximum regions and retain high-confidence candidate points. For each candidate point, a Gaussian weighted window is used to calculate the weighted response value of the surrounding area, further enhancing spatial consistency and eliminating isolated noise points, thus completing heatmap optimization. For the affinity map, multiple locations are uniformly sampled along the limb direction between candidate points, extracting candidate limb vectors and affinity vectors for each point. The average matching degree between the candidate limb vectors and affinity vectors is calculated to verify the rationality of the connections between keypoints. The weighted response value of the heatmap and the affinity matching degree are weighted and fused to obtain a preliminary comprehensive confidence score, which is then normalized to generate a confidence score.

[0102] Based on experimental verification, a confidence threshold was determined. In the training dataset, when the confidence level is ≥0.5, the keypoint detection accuracy exceeds 95%, and the false positive rate is less than 5%. Therefore, the confidence threshold serves as the dividing line between high and low confidence levels. Based on the confidence score, low-confidence keypoints are filtered out. Based on the filtered high-confidence keypoints of the nose and hip, the vertical height difference of the head is calculated. If the confidence level of any keypoint is insufficient, the vertical height difference is set to zero. The mean confidence level of the vertical height difference of the head that meets the confidence threshold is recorded.

[0103] For each limb (left front, right front, left back, right back), verify whether the confidence of the corresponding key points such as shoulder, elbow, hip, and knee meets the confidence threshold. If the confidence of all related key points meets the threshold, calculate the straight-line distance from shoulder to elbow and from hip to knee to measure the extension of unilateral limb. If the confidence of any key point is insufficient, directly determine that the opening of the limb on that side is invalid and assign a value of zero.

[0104] After completing the calculation of one side of the limb, a one-sided weight value reflecting the reliability of the data is generated based on the average confidence of each key point. The higher the weight, the stronger the credibility of the calculation result. The opening degree of the four sides of the limbs and their corresponding weights are combined into an overall limb opening degree measure by weighted averaging. If the sum of the weights of all sides of the limbs is zero (i.e. there is no reliable data), the overall opening degree is determined to be invalid and the result is set to zero.

[0105] Based on the calculated angular deviation between the movement direction angle and the boundary direction angle, and combined with the animal target IDs and movement trajectory queues in the animal target list, a grid coordinate system is established within the protected area. Each grid covers a fixed geographical area. The sequence of trajectory points for each animal target ID is retrieved, and the center coordinates of each trajectory point are mapped to the corresponding grid number. The frequency of the target's stay in each grid is counted. Based on the frequency distribution, the information entropy is calculated, expressed as:

[0106] ;

[0107] in, The entropy value represents the uncertainty of the target's trajectory. A lower entropy value indicates a stronger regularity in the movement path, while a higher entropy value indicates a greater randomness in the behavior. The grid is numbered, meaning each grid represents a fixed area of ​​the protected area. For the goal in the The dwell probability of each grid cell reflects the distribution density of the target in a specific area.

[0108] Based on the information entropy value, the access frequency of each grid per minute is statistically analyzed. Grids with high access frequency are marked as high-risk interest areas and associated with the historical behavioral features of the animal target ID. Velocity sequences, acceleration sequences, angle deviations, vertical height differences, limb spread, and historical behavioral features are combined and aligned by time windows to form a multi-dimensional temporal vector. This vector is then input into a pre-trained LSTM-TCN hybrid model. The LSTM units sequentially process the input multi-dimensional temporal vector, capturing the long-term dependencies of target behavior and outputting 128-dimensional temporal features while retaining time step information. The TCN's dilated causal convolutional layers sequentially apply dilation factors, convolutional kernels, and expanded receptive fields, extracting local behavioral patterns and outputting temporal features with the same dimension as the LSTM output. The output features of the LSTM and TCN are concatenated into a 256-dimensional vector, input into a fully connected layer, and the attack probability is output through the Sigmoid function, representing the likelihood of the target initiating an attack. The Softmax function is then used to map the attack probability to a threat level.

[0109] Specifically, the dynamic division ratio points are set to 0.3 and 0.7, and the attack probability interval is evenly divided into three segments corresponding to low threat level, medium threat level and high threat level respectively. When the attack probability is less than 0.3, it represents low threat level; when the attack probability is less than 0.7 but greater than or equal to 0.3, it represents medium threat level; and when the attack probability is greater than or equal to 0.7, it represents high threat level.

[0110] By associating with real-time environmental data from the surrounding area, the threat level is dynamically adjusted, triggering fuzzy rules.

[0111] Real-time environmental data includes population density and the status of protective facilities.

[0112] If the probability of attack is high and the population density is also high, the threat level increases by one level. If the probability of attack is high and the protective facilities are ineffective, the threat level increases by one level.

[0113] The initial threat level is upgraded based on conditions in the fuzzy rule base. If multiple rules are met simultaneously, the highest level will take effect.

[0114] By integrating multi-source analysis results, a structured list of threat events is generated, including the latest detection box parameters and instantaneous velocity associated with the animal target ID, the threat level is updated in real time based on the fuzzy logic correction results, the confidence level is the attack probability, a list of high-risk grid numbers, and the location coordinates and status of the nearest protective facility.

[0115] Based on the threat level (low, medium, high) in the threat event list, perform a tiered response:

[0116] Low threat level: Extract target detection box coordinates, movement speed and AR label parameters from the threat event list, input the video stream after fusing visible light enhancement features and infrared specific features into the AR rendering engine, overlay dynamic tracking box (red semi-transparent rectangle) and risk label (exclamation mark icon), and push the AR image to the monitoring terminal in real time via WebRTC protocol, and simultaneously record the AR image timestamp, animal target ID and rendering parameters;

[0117] Medium Threat Level: Retrieve the target location and threat level from the event list, trigger the frequency and flashing mode parameters of the audible and visual alarm device, send alarm information through the SMS gateway, including the target location, threat level and timestamp, and write the device ID, command time and response status to the local log;

[0118] High Threat Level: Acquire target coordinates, movement trajectory, and protective facility status. Based on the most recent target coordinate sequence in the movement trajectory queue, predict the trajectory in the near future using cubic spline interpolation. Simultaneously, determine available deterrent devices based on the protective facility status. When the protective facility status is normal, select UAV patrol and generate a patrol path centered on the predicted trajectory point. When the protective facility status is ineffective, activate the ultrasonic transmitter to periodically emit pulsed ultrasonic waves, and activate the lighting jamming system to activate the strobe white LED array, which flashes at fixed time intervals.

[0119] In summary, this invention dynamically adjusts the primary and secondary data streams based on the signal-to-noise ratio detection of edge computing nodes, and combines VSYNC pulse synchronization and clock calibration of FPGA timers to ensure cross-modal data timestamp alignment errors and avoid low-light missed detection problems caused by fixed threshold switching. The visible light frequency domain phase spectrum preserves high-frequency texture details, and the infrared three-level wavelet decomposition extracts diagonal high-frequency subbands. Sparse features are generated by suppressing non-maximum values ​​in the neighborhood through Sobel gradient magnitude and neighborhood, and complementary feature matrices are formed after bilinear interpolation alignment, improving the target discrimination in complex backgrounds. The hybrid feature tensor is horizontally divided into equal-width regions, and the mean values ​​of infrared and visible light features are calculated to generate a set of pseudo-anchor points. The improved YOLOv7 detection head drives similar target features to gather towards pseudo-anchor points through similar aggregation and dissimilar repulsion loss functions, suppressing cross-block interference and improving the detection accuracy of animal targets.

[0120] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent security video monitoring system for preventing animal aggression, characterized in that: The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation.

2. The intelligent security video surveillance system for preventing the attacking behavior of animals as claimed in claim 1, wherein: The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application relates to a multi-spectral feature extraction method and device for multi-modal video stream data packet generation. The application 3. The intelligent security video surveillance system for preventing an animal's aggressive behavior according to claim 1, wherein: The generating multi-scale candidate boxes by improving the YOLOv7 detection head comprises the following steps, The mixed feature tensor is divided into non-overlapping equal-width regions, and the average values of the infrared and visible light feature vectors in the regions are extracted to generate a pseudo-anchor point set; The pseudo-anchor point set is input into the improved YOLOv7 detection head for multi-scale feature map convolution processing to generate multi-scale candidate boxes.

4. The intelligent security video surveillance system for preventing an animal's aggressive behavior according to claim 1, wherein: The pseudo-anchor point guided same-class aggregation and different-class repulsion loss refers to performing same-class aggregation constraint on the infrared specific features of the candidate boxes to drive the same-class features to gather around the pseudo-anchor points, and performing different-class repulsion constraint on the left and right adjacent blocks of the block where the candidate boxes are located to suppress cross-block feature confusion.

5. The intelligent security video surveillance system for preventing an animal's aggressive behavior according to claim 1, wherein: The generating animal target list comprises the following steps, The boundary box parameters of each candidate box in the high-confidence candidate box set are converted into state vectors, and the infrared pseudo-anchor point features and the visible light enhanced pseudo-anchor point features are associated; The current target position is predicted based on the state vector, and the state vector is updated by Mahalanobis distance matching to generate a motion-corrected prediction box; The corrected prediction box is cross-frame associated with the infrared pseudo-anchor point features and the visible light enhanced pseudo-anchor point features, the spatio-temporal-feature comprehensive similarity is calculated, the continuously unmatched target is determined as leaving the monitoring area and removed from the tracking list, and the animal target list is generated.

6. The intelligent security video surveillance system for preventing an animal's aggressive behavior according to claim 1, wherein: The outputting attack probability and mapping to dynamic threat level comprises the following steps, Based on the angle deviation of the motion direction angle in the animal target list and the direction angle of the protected area boundary, the head height difference and the limb opening degree detected by OpenPose, a multi-dimensional time sequence vector is formed combining information entropy; The multi-dimensional time sequence vector is input into an LSTM-TCN hybrid model, the attack probability is output, and the low, medium and high threat levels are divided by proportional point.

7. The intelligent security video surveillance system for preventing an animal's aggressive behavior according to claim 1, wherein: The triggering hierarchical response mechanism refers to superimposing a dynamic tracking box by an AR rendering engine at a low threat level and pushing it to a monitoring terminal in real time, triggering sound and light alarms and sending alarm information to an SMS gateway at a medium threat level, and activating a drone cruise and ultrasonic interference based on trajectory prediction at a high threat level.

Images