Intelligent video monitoring method and system based on multi-source data fusion

Abstract

The invention provides an intelligent video monitoring method and system based on multi-source data fusion, and belongs to the technical field of video data analysis. Firstly, multi-source video data are obtained and preprocessed; inputting the preprocessed video data into a spatial-temporal feature coding and decoding network to obtain a preliminary result of pedestrian image abnormal behavior detection; based on the obtained preliminary detection result, pedestrian images in the abnormal behaviors are extracted, and abnormal behavior image data are obtained; inputting the obtained abnormal behavior image data into a feature recognition network to obtain a feature vector; and finally, inputting the obtained feature vectors into a cross-modal pedestrian feature registration network based on double-alignment feature embedding to obtain a final pedestrian image abnormal behavior detection result. The abnormal event detection method based on multi-source data fusion is efficient and stable, multi-source monitoring data can be fused, the method adapts to changes of different scenes and illumination conditions, and sudden abnormal behaviors in the monitoring scenes are recognized in time.

Description

Technical Field

[0001] The present invention belongs to the technical field of video data analysis, and in particular relates to an intelligent video monitoring method based on multi-source data fusion. Background Art

[0002] Video surveillance is an important information auxiliary means in the field of daily public safety. The city-level surveillance camera network is constantly expanding to fully cover every corner of the city and promptly detect and respond to potential security threats. With the surge in the number of surveillance cameras, on the one hand, the spatial distribution of the surveillance camera network is wide, and there may be diverse lighting conditions and background environments in different scenes. On the other hand, the images collected by various types of cameras in different working time periods show different modal characteristics. The existing technology has the following limitations: Single-modal data analysis: Intelligent video surveillance needs to process multi-source video data, including RGB, grayscale, near-infrared, and thermal infrared. However, current intelligent video surveillance methods mainly focus on the modeling and analysis of single-modal data, usually using only RGB or grayscale images. This limitation performs poorly in conditions such as lighting changes, scene complexity, and visual occlusion, resulting in a decrease in the accuracy of abnormal event detection and pedestrian re-identification. The information provided by different modal data (such as near-infrared, thermal infrared, etc.) is complementary, and the fusion of these multi-source data can significantly improve the robustness and adaptability of the monitoring system; Mixed spatial and temporal information: Multi-source video data contains both spatial and temporal information, but existing research methods have not fully considered the modeling of normal video data distribution using temporal-spatial mixed information. This difference may cause existing abnormal event detection models to be less sensitive when dealing with abnormal events that are not much different from normal events; Cross-modal person re-identification: For the problem of cross-modal person re-identification under multi-source data, existing work mainly uses the nonlinear fitting ability of deep neural networks to transform the features of different modalities into the same feature subspace to extract discriminative pedestrian feature representations. However, this method fails to effectively unify the feature extraction and metric learning strategies of multi-modal video data. In addition, when processing the features of cross-modal pedestrian images with spatial and modal misalignment characteristics, it is often difficult to achieve effective registration, resulting in insufficient robustness in multi-source data scenarios.

[0003] Therefore, there is an urgent need for an efficient and stable abnormal event detection algorithm based on multi-source data fusion, which can integrate multi-source monitoring data and adapt to changes in different scenes and lighting conditions, and promptly identify sudden abnormal events in the monitoring scenes and issue alarms. Summary of the invention

[0004] In view of the above problems, the first aspect of the present invention provides an intelligent video monitoring method based on multi-source data fusion, comprising the following steps: S1, obtain multi-source video data and pre-process the data; S2, input the preprocessed video data into the spatiotemporal feature encoding and decoding network to obtain the preliminary results of abnormal behavior detection of pedestrian images; S3, based on the preliminary results of abnormal behavior detection of pedestrian images obtained in S2, extract the pedestrian images in abnormal behavior to obtain abnormal behavior image data; S4, inputting the abnormal behavior image data obtained in S3 into a feature recognition network to obtain a feature vector; the feature recognition network includes a feature learning module and a feature mapping module, and the pedestrian image is input into the feature learning module to obtain the local features of the image , local features Input to the feature mapping module to obtain the hypersphere domain mapping features ; The feature recognition network is trained based on a metric learning strategy of hypersphere manifold and KL divergence; S5, the feature vector obtained in S4 is input into the cross-modal pedestrian feature registration network based on dual alignment feature embedding. First, the backbone network and global average pooling layer are used to extract The fine-grained features are then used, and the convolution layer and batch normalization layer are used to reduce the dimension of the fine-grained features. Finally, the fine-grained features after dimensionality reduction are input into the fully connected layer to obtain the final pedestrian image abnormal behavior detection results.

[0005] Preferably, the multi-source video data in S1 includes visible light pedestrian video data and infrared light pedestrian video data, the multi-source video data is standardized and denoised, and the spatial resolution and brightness are uniformly adjusted to obtain pre-processed multi-source video data.

[0006] Preferably, the spatiotemporal feature encoding and decoding network in S2 includes a 3D encoding module, a 2D decoding module, a resampling module, a temporal downsampling module and a spatiotemporal consistency discriminator, which converts the visible light video data and infrared video data The specific steps of inputting the spatiotemporal feature encoding and decoding network to obtain the preliminary results of abnormal behavior detection are as follows: S21, Pedestrian Video and Uniformly sample to k frames, and form a video sequence consisting of k video frames Input to the spatiotemporal feature encoding and decoding network, where Indicates the first Frame image; S22, the hidden layer feature vector obtained by inputting the five 3D encoding modules connected in series is calculated as follows: ; ; ; ; ; in , , , and Respectively represent a first 3D encoding module, a second 3D encoding module, a third 3D encoding module, a fourth 3D encoding module and a fifth 3D encoding module; , , and are output features of the first 3D encoding module, the second 3D encoding module, the third 3D encoding module, and the fourth 3D encoding module respectively; S23, will Input resampling module, Resample the hidden space to obtain new features , the specific calculation formula is as follows: ; in, is a random vector between zero and one, and They are The mean and variance of the standard normal distribution in high-dimensional space; and Learned by two fully connected layers The data distribution is obtained; During the sampling process, Kullback Leibler discreteness is used as a constraint for resampling to ensure the characteristics after resampling With the original features They obey the same distribution, where the calculation formula of Kullback Leibler discreteness is as follows: ; in, represents the Kullback Leibler discreteness, is the Kullback Leibler discrete value; S24, will After inputting the five 2D decoding modules connected in series, it is decoded into the next frame image of the input sequence. , the specific calculation formula is as follows: ; ; ; ; ; in, , , , and Respectively represent a first 2D decoding module, a second 2D decoding module, a third 2D decoding module, a fourth 2D decoding module and a fifth 2D decoding module; , , and are output features of the first 2D decoding module, the second 2D decoding module, the third 2D decoding module, and the fourth 2D decoding module respectively; is the next frame image of the generated input sequence; represents the time domain downsampling module, Represents the feature fusion function, which directly merges two tensors with the same height and width in the channel dimension; the convolution operations in the 2D decoding module are all two-dimensional convolutions; S25, and After inputting the spatiotemporal consistency discriminator at the same time, we can obtain the preliminary abnormal behavior detection results. .

[0007] Preferably, the spatiotemporal feature encoding and decoding network is optimized and trained by a spatiotemporal consistency training strategy, and the cost function of the spatiotemporal consistency training strategy is composed of a pixel-level error square function and an adversarial loss function; The pixel-level square error function , the entire spatiotemporal feature encoding and decoding network is trained by minimizing the following cost function: ; in, is the height and width of the image in the input video sequence. This function calculates the result pixel by pixel. Its true value The square of the error between It is the symbol for modulo calculation; The adversarial loss function is composed of a generation loss function and a discrimination loss function; Generating loss function for: ; Discriminative loss function for: ; in, represents a matrix of all zeros, Pedestrian image samples generated by decoder G in the spatiotemporal feature encoding and decoding network , represents the concatenation of time channels, Represents dimension and consistent identity matrix; is the actual video frame sequence, the corresponding real video frame and the decoder input to the spatiotemporal feature encoding and decoding network By cascading on the time channel, we can get , is the sampling value of the real video sequence, ; The training process of the spatiotemporal consistency enhancement network uses the decoder in the spatiotemporal feature encoding and decoding network and the spatiotemporal consistency discriminator to alternately train until both reach convergence.

[0008] Preferably, the feature recognition network in S4 includes a feature learning module and a feature mapping module, wherein the feature learning module is composed of a convolutional neural network, a fully connected layer, and a batch normalization layer; the feature mapping module is composed of a random inactivation layer, a fully connected layer, and a two-norm normalization layer; the specific steps of implementing feature extraction of pedestrian abnormal behavior images by the feature recognition network are as follows: The pedestrian image obtained by S3 Input feature learning module to obtain local features of the image , the specific calculation formula is as follows: ; in is the convolutional layer, is the fully connected layer, is the batch normalization layer; The local features Input feature mapping module to obtain hypersphere domain mapping features , the specific calculation formula is as follows: ; in is the random dropout layer, is a two-norm normalization layer; through the mapping of the feature mapping module, the domain detailed features of different modes are mapped to the same hyperspherical manifold to obtain the domain shared features on the hyperspherical manifold.

[0009] Preferably, the feature recognition network in S4 is trained based on a metric learning strategy of hypersphere manifold and KL divergence, specifically: First, the hypersphere manifold loss function is used to train the feature recognition network to accelerate the convergence process of the model. The hypersphere manifold loss function is as follows: ; Among them, s is the proportional coefficient of the cosine value of the angle cos, which is used to accelerate the training process of the model, and C represents the number of abnormal behavior labels of pedestrians. is the number of pedestrian images contained in a batch; Represents the vector angle between different behavior categories in different pedestrian images; Indicates The angle between the abnormal behavior prediction output vector of a pedestrian image and the true label vector; Then, a KL divergence constraint training feature recognition network is constructed; that is, a probability constraint method based on Kullback-Leibler discreteness is used to measure the probability prediction value of different modalities of the same person. and The similarity between them improves the modality independence of cross-modal features. arrive The KL divergence value of Calculated by the following formula: ; Since the KL divergence is asymmetric, arrive Divergence for: ; in, is the pedestrian behavior prediction result under visible light mode, Pedestrian behavior prediction results in infrared light modality.

[0010] Preferably, the specific processing process of the cross-modal pedestrian feature registration network based on dual alignment feature embedding in S5 is: S51, will Input ResNet50 backbone network; the network before the last pooling layer of traditional ResNet50 is used as ResNet50 backbone network. Input the ResNet50 backbone network and output a multi-channel feature map M. The specific calculation formula is as follows: ; Where M is a three-dimensional tensor; S52, the feature map The image is randomly divided into K non-overlapping parts along the vertical direction, and then global average pooling is performed on each part to obtain fine-grained features. , for the Part , the calculation formula is as follows: ; in is a feature vector of 2048 dimensions; S53, the fine-grained features The input convolutional layer and batch normalization layer reduce The dimension is reduced to a feature vector with a dimension of 256*K; S54, respectively inputting the K feature vectors of the fine-grained features after dimensionality reduction into K fully connected layers with different weights to obtain the final pedestrian image abnormal behavior detection result.

[0011] Preferably, the cross-modal pedestrian feature registration network based on dual alignment feature embedding is trained by a modality consistency metric learning strategy, including an intra-class distribution constraint loss function and an inter-class correlation constraint loss function, as follows: The intra-class distribution constraint loss function is minimized by minimizing the following formula Discreteness, obtain the intra-class distribution constraint loss function : ; in and are the means of the visible light modality pedestrian features and the infrared modality pedestrian features, and is the standard deviation of the visible light modality pedestrian features and the infrared modality pedestrian features, , , and The corresponding calculation formulas are as follows: ; ; ; ; in, Indicates The features of infrared pedestrian images, Indicates Features of pedestrian images in visible light mode; The inter-class correlation constraint loss function The specific calculation formula is as follows: ; in is the Pearson correlation coefficient matrix of a batch of infrared images The elements, is the Pearson correlation coefficient matrix of a batch of visible light images The elements, and The calculation formula is as follows: ; .

[0012] The second aspect of the present invention provides an intelligent video monitoring system based on multi-source data fusion, which is applied to the intelligent video monitoring method as described in the first aspect, including: a video acquisition module and a backend server; The video acquisition module is used to acquire multi-source video data; The backend server is internally deployed with a video preprocessing module, a spatiotemporal feature encoding and decoding network, an abnormal behavior extraction module, a feature recognition network, and a cross-modal pedestrian feature registration network based on dual-aligned feature embedding; The video preprocessing module is used to standardize and remove noise from multi-source videos, and uniformly adjust spatial resolution and brightness; The spatiotemporal feature encoding and decoding module is used to input the preprocessed video data to obtain preliminary results of abnormal behavior detection of pedestrian images; An abnormal behavior extraction module is used to extract pedestrian images in abnormal behavior and obtain abnormal behavior image data; The feature recognition network is used to obtain a feature vector; The cross-modal pedestrian feature registration network based on dual-aligned feature embedding is used to obtain the final pedestrian image abnormal behavior detection result.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. In response to the problem of mixed spatial and temporal information under multi-source data, the present invention proposes an abnormal event detection method based on enhanced spatiotemporal consistency, which improves the spatiotemporal consistency of future frame prediction of video images on the training data set through a spatiotemporal feature encoding and decoding network and a spatiotemporal consistency discrimination network. The difference in prediction results between abnormal data and normal data is improved to more significantly detect abnormal events in video sequences. Specifically, the present invention proposes a framework for predicting future frames of an input frame sequence based on a deep convolutional network, and extracts the spatiotemporal mixed information of the input video sequence through a three-dimensional convolution operation. At the same time, the present invention proposes a spatiotemporal consistency between the future frame images of the video generated by the spatiotemporal consistency discrimination network and the input video sequence. Combining the above two methods, the disturbance of abnormal events on the model output can be effectively increased, thereby differentiating the model's response to abnormal and normal data; 2. In view of the difficulty in effectively unifying the feature extraction and metric learning strategies of multimodal video data in cross-modal pedestrian re-identification, this paper proposes a cross-modal pedestrian re-identification method based on hypersphere manifold embedding. By using hypersphere manifold embedding representation and bidirectional sorting loss, the dimensions of the objective functions in the two tasks of recognition and classification are unified, thereby extracting highly discriminative features while ensuring the high coupling of the feature extraction process and metric constraints. In addition, this paper further analyzes the hypersphere manifold embedding and proposes a decorrelation method based on singular value decomposition to further expand the inter-class distance of features on the hypersphere manifold; 3. In order to solve the problem of difficult to effectively align cross-modal pedestrian image features in pedestrian re-identification from multi-source video data, the present invention proposes a cross-modal pedestrian feature alignment method based on dual-aligned feature embedding. By using fine-grained feature extraction and modal consistency constraints to solve the spatial misalignment and modal misalignment problems of cross-modal pedestrian images, the two misalignment problems of cross-modal pedestrian re-identification are simultaneously solved in a unified framework, so that the model has better robustness to pedestrian posture while extracting highly discriminative feature representations with modal invariance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The figure is an overall flow chart of the intelligent video monitoring method based on multi-source data fusion of the present invention.

[0015] Figure 2 Schematic diagram of the spatiotemporal feature encoding and decoding network structure of the present invention.

[0016] Figure 3 Schematic diagram of the feature recognition network structure of the present invention.

[0017] Figure 4 Schematic diagram of the cross-modal pedestrian feature registration network structure based on dual alignment feature embedding in the present invention.

[0018] Figure 5 This is a convergence curve diagram of the training loss function during the experimental process of the present invention. DETAILED DESCRIPTION

[0019] The invention will be further described below in conjunction with specific embodiments.

[0020] The overall logic of the solution of the present invention is as follows Figure 1As shown. First, obtain multi-source video data and preprocess the data; input the preprocessed video data into the spatiotemporal feature encoding and decoding network to obtain preliminary results of abnormal behavior detection of pedestrian images; based on the obtained preliminary results of abnormal behavior detection of pedestrian images, extract pedestrian images in abnormal behavior to obtain abnormal behavior image data; input the obtained abnormal behavior image data into the feature recognition network to obtain feature vectors; the feature recognition network includes a feature learning module and a feature mapping module, the pedestrian image is input into the feature learning module to obtain the local features of the image, and the local features are input into the feature mapping module to obtain the hypersphere domain mapping features; finally, the obtained feature vector is input into the cross-modal pedestrian feature registration network based on dual alignment feature embedding to obtain the final pedestrian image abnormal behavior detection results.

[0021] First, we collect visible light pedestrian video data and infrared light pedestrian video data to form multi-source video data. Secondly, we standardize and denoise the multi-source video data, and uniformly adjust the spatial resolution and brightness to obtain pre-processed multi-source video data. Indicates that infrared video data is used express.

[0022] 2. Build a spatiotemporal feature encoding and decoding network to obtain visible light video data and infrared video data The spatiotemporal feature encoding and decoding network is input into the network to obtain the preliminary results of abnormal behavior detection in pedestrian images; secondly, a spatiotemporal consistency training strategy is proposed to train the spatiotemporal feature encoding and decoding network.

[0023] 1. About the empty feature encoding and decoding network: The spatiotemporal feature encoding and decoding network structure is as follows Figure 2 As shown in Figure 2, the network consists of a 3D encoding module, a 2D decoding module, a resampling module, a temporal downsampling module, and a spatiotemporal consistency discriminator. and infrared video data The specific steps of inputting the spatiotemporal feature encoding and decoding network to obtain the preliminary results of abnormal behavior detection are as follows: Pedestrian video and Uniformly sample to k frames, and form a video sequence consisting of k video frames Input to the spatiotemporal feature encoding and decoding network, where Indicates the first Frame image; Will After inputting 5 3D encoding modules connected in series, we get The hidden feature vector of , the specific calculation formula is as follows: ; ; ; ; ; in , , , and Respectively represent a first 3D encoding module, a second 3D encoding module, a third 3D encoding module, a fourth 3D encoding module and a fifth 3D encoding module; , , and are the output features of the first 3D encoding module, the second 3D encoding module, the third 3D encoding module and the fourth 3D encoding module respectively; the input of the 3D encoding module of the present invention is a five-dimensional tensor, so the spatiotemporal mixed information contained in the pedestrian video can be well captured, that is, the feature Represents the spatiotemporal fusion information contained in pedestrian videos; Will Input resampling module, Resample the hidden space to obtain new features , the specific calculation formula is as follows: ; in, is a random vector between zero and one, and They are The mean and variance of the standard normal distribution in high-dimensional space; and Learned by two fully connected layers The data distribution is obtained; During the sampling process, Kullback Leibler discreteness is used as a constraint for resampling to ensure the characteristics after resampling With the original features They obey the same distribution, where the calculation formula of Kullback Leibler discreteness is as follows: ; in, represents the Kullback Leibler discreteness, is the Kullback Leibler discrete value; since the training process uses normal video data, this part of the resampling module will only make correct parameter estimates for the distribution of the features that contain the spatiotemporal information of normal events; through this resampling process, the difficulty of the future frame prediction task is increased. When the learning ability of the model is the same, adding an additional resampling module can effectively reduce the possibility of the model generalizing the future frame prediction ability from normal video data to abnormal video data, thereby enhancing the disturbance of the abnormal video data to the model; Will After inputting the five 2D decoding modules connected in series, it is decoded into the next frame image of the input sequence. , the specific calculation formula is as follows: ; ; ; ; ; in, , , , and Respectively represent a first 2D decoding module, a second 2D decoding module, a third 2D decoding module, a fourth 2D decoding module and a fifth 2D decoding module; , , and are output features of the first 2D decoding module, the second 2D decoding module, the third 2D decoding module, and the fourth 2D decoding module respectively; is the next frame image of the generated input sequence; represents the time domain downsampling module, represents a feature fusion function, which directly merges two tensors with the same height and width in the channel dimension; the convolution operations in the 2D decoding module constructed by the present invention are all two-dimensional convolutions, and the model parameters are shown in Table 1; Will and After inputting the spatiotemporal consistency discriminator at the same time, we can obtain the preliminary abnormal behavior detection results. .

[0024] Table 1 2D decoding module parameter table

[0025] 2. About the spatiotemporal consistency training strategy: The present invention proposes a spatiotemporal consistency training strategy to train the spatiotemporal feature encoding and decoding network to ensure that the 3D encoding module and the 2D decoding module in the spatiotemporal feature encoding and decoding network can fully learn the spatiotemporal consistency features between each frame of the pedestrian video. The cost function of the spatiotemporal consistency training strategy consists of a pixel-level error square function and an adversarial loss function, which is as follows: Pixel-level squared error function , the entire spatiotemporal feature encoding and decoding network is trained by minimizing the following cost function: ; in, is the height and width of the image in the input video sequence. This function calculates the result pixel by pixel. Its true value The square of the error between It is the symbol for modulo calculation; Adversarial loss function, which consists of generation loss function and discriminative loss function; Generating loss function for: ; Discriminative loss function for: ; in, represents a matrix of all zeros, Pedestrian image samples generated by decoder G in the spatiotemporal feature encoding and decoding network , represents the concatenation of time channels, Represents dimension and consistent identity matrix; is the actual video frame sequence, the corresponding real video frame and the decoder input to the spatiotemporal feature encoding and decoding network By cascading on the time channel, we can get , is the sampling value of the real video sequence, ; For the spatiotemporal consistency discriminator, when the input of the spatiotemporal consistency discriminator is When , the spatiotemporal consistency discriminator needs to output a feature map with all values ​​1, which means that the discriminator will input the video sequence is determined to be a sequence of temporal and spatial consistency. or When , the expected output of the discriminator is a feature map of all 0s, which means that the discriminator will input the video sequence and It is judged to be inconsistent in time and space. Therefore, the loss function used to train the discriminator can be written as: ; The training process of the spatiotemporal consistency enhancement network uses the generator and discriminator to alternately train until both reach convergence: A: Fix the parameters of the spatiotemporal consistency discrimination network and train the spatiotemporal feature encoding and decoding network. The objective function during training is: ; in, , is a trade-off factor used to balance the multi-task objective function; for The divergence of B: Fix the parameters of the spatiotemporal feature encoding and decoding network, train the spatiotemporal consistency discrimination network, and the objective function during training is .

[0026] 3. Preliminary results of abnormal behavior detection based on pedestrian images , will obtain and Extract the portrait images of pedestrians with abnormal behavior and obtain abnormal behavior image data First, and The images are divided into frames to obtain a pedestrian image dataset; then, according to R, it is determined which images in the pedestrian image dataset contain pedestrians with abnormal behaviors; finally, the images containing pedestrians with abnormal behaviors in the pedestrian image dataset are screened out.

[0027] 4. Construct a feature recognition network. The feature recognition network is as follows Figure 3 As shown, the obtained abnormal behavior image is input into the feature recognition network to obtain the feature vector; secondly, a metric learning strategy based on hypersphere manifold and KL divergence is proposed to train the feature recognition network.

[0028] 1. About feature recognition network: Construct a feature recognition network to realize feature extraction of pedestrian abnormal behavior images; the feature recognition network includes a feature learning module and a feature mapping module, wherein the feature learning module is composed of a convolutional neural network, a fully connected layer, and a batch normalization layer; the feature mapping module is composed of a random inactivation layer, a fully connected layer, and a two-norm normalization layer; the specific steps of realizing feature extraction of pedestrian abnormal behavior images by the feature recognition network are as follows: The pedestrian image obtained Input feature learning module to obtain local features of the image , the specific calculation formula is as follows: ; in is the convolutional layer, is the fully connected layer, is the batch normalization layer; The local features Input feature mapping module to obtain hypersphere domain mapping features , the specific calculation formula is as follows: ; in is the random dropout layer, is a two-norm normalization layer; through the mapping of the feature mapping module, the domain detailed features of different modes are mapped to the same hyperspherical manifold, and the domain shared features on the hyperspherical manifold are obtained;

[0029] 2. Metric learning strategy based on hypersphere manifold and KL divergence: The specific steps are as follows: The present invention first uses a hypersphere manifold loss function to train the feature recognition network to accelerate the convergence process of the model. The hypersphere manifold loss function is as follows: ; Among them, s is the proportional coefficient of the cosine value of the angle cos, which is used to accelerate the training process of the model, and C represents the number of abnormal behavior labels of pedestrians. is the number of pedestrian images; Then, a KL divergence constraint is constructed to further train the feature recognition network; the present invention uses a probability constraint method based on Kullback-Leibler discreteness to measure the probability prediction value of different modalities of the same person and The similarity between them can further improve the modality independence of cross-modal features. Assume that a batch of two different input modalities contains N samples. Then arrive The KL divergence value can be calculated by the following formula: ; Since the KL divergence is asymmetric, arrive The divergence of is: ; in, is the pedestrian behavior prediction result under visible light mode, Pedestrian behavior prediction results in infrared light modality.

[0030] 5. Construct a cross-modal pedestrian feature registration network based on dual alignment feature embedding. The network structure is as follows: Figure 4As shown in the figure, the obtained feature vector is input into the cross-modal pedestrian feature registration network based on dual-aligned feature embedding to obtain the final pedestrian image abnormal behavior detection result; secondly, a modal consistency metric learning strategy is proposed to train the cross-modal pedestrian feature registration network based on dual-aligned feature embedding.

[0031] 1. Cross-modal pedestrian feature registration network based on dual alignment feature embedding: The present invention constructs a cross-modal pedestrian feature registration network based on dual-aligned feature embedding to obtain the final pedestrian image abnormal behavior detection result; the cross-modal pedestrian feature registration network based on dual-aligned feature embedding includes a ResNet50 backbone network, a global average pooling layer, a convolutional layer, a batch normalization layer and a fully connected layer. The ResNet50 backbone network and the global average pooling layer are used to extract The fine-grained features are then used in the convolution layer and batch normalization layer to reduce the dimension of the fine-grained features, reducing them to feature vectors with a dimension of 256. Finally, the fine-grained features after dimensionality reduction are input into the fully connected layer to obtain the final pedestrian image abnormal behavior detection results. The specific steps are as follows: 1) Input ResNet50 backbone network; the present invention adopts the network before the last pooling layer of ResNet50 as the ResNet50 backbone network. Input the ResNet50 backbone network and output a multi-channel feature map M. The specific calculation formula is as follows: ; Where M is a three-dimensional tensor; 2) The feature map The image is randomly divided into K non-overlapping parts along the vertical direction, and then global average pooling is performed on each part to obtain fine-grained features. For the Part , the calculation formula is as follows: ; in is a feature vector of 2048 dimensions; 3) Fine-grained features The input convolutional layer and batch normalization layer reduce The dimension is reduced to a feature vector with a dimension of 256*K; 4) The K feature vectors of the fine-grained features after dimensionality reduction are input into K fully connected layers with different weights to obtain the pedestrian abnormal behavior detection results.

[0032] 2. Modal consistency metric learning strategy: The present invention trains a cross-modal pedestrian feature registration network based on dual alignment feature embedding through a modality consistency metric learning strategy to improve the cross-modal recognition accuracy of the network; the modality consistency metric learning strategy consists of an intra-class distribution constraint loss function and an inter-class correlation constraint loss function, which are as follows: The proposed cross-modal alignment is achieved through modality consistency metric constraints, which mainly consists of two parts, namely, the intra-class distribution constraint loss function and the inter-class correlation constraint loss function; 1) Constructing intra-class distribution constraint loss function: Pedestrian features of different modes , which conforms to the following distribution representation: ; Here, V and I are used to represent the visible light modality and infrared modality respectively. For a pedestrian with class label i, since its visible light modality image and infrared modality image are different forms of the pedestrian as an objective subject in different modality representation spaces, and the features after the feature extraction network should be a high-level cognitive level feature, it is hoped that the feature distribution of the visible light image of the same pedestrian should be as close as possible to the feature distribution of its infrared image. Jensen-Shannon discreteness is used to measure the difference between the two distributions. and Similarity: ; in Is a mixed distribution , Denotes the Kullback-Leibler dispersion. The Jensen Shannon dispersion of the two distributions can also be measured by Jeffreys similarity. To calculate, then and The similarity between them can be expressed as: ; The above formula can be rewritten as: ; Minimization is achieved directly by minimizing the following formula Discreteness, obtain the intra-class distribution constraint loss function: ; in and Characteristics and The mean of and The covariance matrix is and The corresponding calculation formula is as follows: ; 2) Construct inter-class correlation constraint loss function: The Pearson Correlation matrix is ​​used as a correlation representation of image features in a visible light batch. For two n-dimensional feature vectors and , their Pearson correlation coefficient can be expressed as: ; The accumulated features in a visible light batch are recorded as , and the corresponding cumulative features in the infrared light batch are recorded as , where N is the number of pedestrian images contained in a batch of each modality during training. By calculating the feature between each two images in a visible light batch The Pearson correlation coefficient matrix of all images in a batch of visible light can be obtained: ; in, Represents the first image feature in a batch of visible light Features and Features The correlation coefficient of a batch of infrared images can be obtained by the same method: ; The two correlation coefficient matrices can be rewritten as follows using a matrix column vector group: ; By constructing a related loss function to constrain the matrix and The consistency obtains the inter-class correlation constraint loss function, the formula is: ; in is the Pearson correlation coefficient matrix of a batch of infrared images The elements, is the Pearson correlation coefficient matrix of a batch of visible light images The elements.

[0033] 6. Model deployment: First, the trained spatiotemporal feature encoding and decoding network, feature recognition network and cross-modal pedestrian feature registration network based on dual-aligned feature embedding are deployed on the back-end server; secondly, the infrared pedestrian video data collected by the infrared camera and the visible light pedestrian video data collected by the visible light camera are input into the back-end server for video sampling and frame processing to obtain frame-by-frame pedestrian images; finally, the pedestrian images are input into the spatiotemporal feature encoding and decoding network to obtain the preliminary recognition results of pedestrian abnormal behavior, the images containing pedestrians with abnormal behavior are input into the feature recognition network to obtain the feature vector, and the feature vector is input into the cross-modal pedestrian feature registration network based on dual-aligned feature embedding to obtain the final pedestrian abnormal behavior detection results.

[0034] VII. Experimental results of abnormal behavior detection using multi-source video data under enhanced spatiotemporal consistency: Experimental dataset: CUHKAvenue: This dataset contains 16 videos for training and 21 videos for testing. The length of each video is about 1 minute. UCSDPedestrian: This dataset consists of two subsets, UCSD Ped1 and UCSD Ped2, and existing methods are performed independently on these two sets. Among them, UCSD Ped1 includes 34 short video sequences for training and 36 video sequences containing 40 abnormal events for testing, each video sequence is 200 frames long. UCSD Ped2 includes 16 training videos and 12 test videos with 12 abnormal events, each video has about 170 frames. The regular events in these two subsets are people walking on the sidewalk, and the abnormal events include bicycles, wheelchairs, vehicles, and people playing skateboards in the crowd.

[0035] ShanghaiTech Campus (SHTech): This dataset includes more than 270,000 training frames and more than 42,000 test frames, of which 17,000 frames are abnormal frames. This dataset covers 13 different scenarios.

[0036] Table 2. Abnormal event recognition accuracy on the experimental dataset

[0037] As shown in Table 2, on the CUHKAvenue data, the proposed method of the present invention improves the detection accuracy of abnormal time by nearly 6 percentage points compared to the STAE-O method using additional optical flow information. On the Ped2 database, the proposed method of the present invention improves the detection effect by 2 percentage points compared to the best method Frame-Pred. From the experimental results in the table, the proposed method of the present invention surpasses the best existing algorithms on the three widely used datasets.

[0038] In order to verify the effectiveness and robustness of the method of the present invention, the present invention selected the ShanghaiTech Campus (SHTech) dataset to conduct 6 pedestrian abnormal behavior detection experiments, where the horizontal axis is the training round, the vertical axis represents the loss function value, and each curve represents an experiment. The total loss function of the 6 experimental model training process converges as follows Figure 5 As shown in the figure, it can be seen that the training convergence speed of the method proposed in the present invention on the six data sets is faster, indicating that the model proposed in the present invention can better learn the fusion features of visible light images and infrared light images, and the training strategy proposed in the present invention can accelerate the convergence of the model.

[0039] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0040] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An intelligent video monitoring method based on multi-source data fusion, characterized in that: The following steps are involved: S1, obtain multi-source video data and pre-process the data; S2, input the preprocessed video data into the spatiotemporal feature encoding and decoding network to obtain the preliminary results of abnormal behavior detection of pedestrian images; S3, based on the preliminary results of abnormal behavior detection of pedestrian images obtained in S2, extract the pedestrian images in abnormal behavior to obtain abnormal behavior image data; S4, inputting the abnormal behavior image data obtained in S3 into a feature recognition network to obtain a feature vector; the feature recognition network includes a feature learning module and a feature mapping module, and the pedestrian image is input into the feature learning module to obtain the local features of the image , local features Input to the feature mapping module to obtain the hypersphere domain mapping features ; The feature recognition network is trained based on a metric learning strategy of hypersphere manifold and KL divergence; S5, the feature vector obtained in S4 is input into the cross-modal pedestrian feature registration network based on dual alignment feature embedding. First, the backbone network and global average pooling layer are used to extract The fine-grained features are then used, and the convolution layer and batch normalization layer are used to reduce the dimension of the fine-grained features. Finally, the fine-grained features after dimensionality reduction are input into the fully connected layer to obtain the final pedestrian image abnormal behavior detection results.

2. The intelligent video monitoring method based on multi-source data fusion according to claim 1, characterized in that: The multi-source video data in S1 includes visible light pedestrian video data and infrared light pedestrian video data. The multi-source video data is standardized and denoised, and the spatial resolution and brightness are uniformly adjusted to obtain pre-processed multi-source video data.

3. The intelligent video monitoring method based on multi-source data fusion according to claim 1, characterized in that: The spatiotemporal feature encoding and decoding network in S2 includes a 3D encoding module, a 2D decoding module, a resampling module, a temporal downsampling module and a spatiotemporal consistency discriminator. and infrared video data The specific steps of inputting the spatiotemporal feature encoding and decoding network to obtain the preliminary results of abnormal behavior detection are as follows: S21, Pedestrian Video and Uniformly sample to k frames, and form a video sequence consisting of k video frames Input to the spatiotemporal feature encoding and decoding network, where Indicates the first Frame image; S22, will After inputting 5 3D encoding modules connected in series, we get The hidden feature vector of , the specific calculation formula is as follows: ； ； ； ； ； in , , , and Respectively represent a first 3D encoding module, a second 3D encoding module, a third 3D encoding module, a fourth 3D encoding module and a fifth 3D encoding module; , , and are output features of the first 3D encoding module, the second 3D encoding module, the third 3D encoding module, and the fourth 3D encoding module respectively; S23, will Input resampling module, Resample the hidden space to obtain new features , the specific calculation formula is as follows: ； in, is a random vector between zero and one, and They are The mean and variance of the standard normal distribution in high-dimensional space; and Learned by two fully connected layers The data distribution is obtained; During the sampling process, Kullback Leibler discreteness is used as a constraint for resampling to ensure the characteristics after resampling With the original features They obey the same distribution, where the calculation formula of Kullback Leibler discreteness is as follows: ； in, represents the Kullback Leibler discreteness, is the Kullback Leibler discrete value; S24, will After inputting the five 2D decoding modules connected in series, it is decoded into the next frame image of the input sequence. , the specific calculation formula is as follows: ； ； ； ； ； in, , , , and Respectively represent a first 2D decoding module, a second 2D decoding module, a third 2D decoding module, a fourth 2D decoding module and a fifth 2D decoding module; , , and are output features of the first 2D decoding module, the second 2D decoding module, the third 2D decoding module, and the fourth 2D decoding module respectively; is the next frame image of the generated input sequence; represents the time domain downsampling module, Represents the feature fusion function, which directly merges two tensors with the same height and width in the channel dimension; the convolution operations in the 2D decoding module are all two-dimensional convolutions; S25, and After inputting the spatiotemporal consistency discriminator at the same time, we can obtain the preliminary abnormal behavior detection results. .

4. The intelligent video monitoring method based on multi-source data fusion according to claim 1, characterized in that: The spatiotemporal feature encoding and decoding network is optimized and trained by a spatiotemporal consistency training strategy, and the cost function of the spatiotemporal consistency training strategy is composed of a pixel-level error square function and an adversarial loss function; The pixel-level square error function , the entire spatiotemporal feature encoding and decoding network is trained by minimizing the following cost function: ； in, is the height and width of the image in the input video sequence. This function calculates the result pixel by pixel. Its true value The square of the error between It is the symbol for modulo calculation; The adversarial loss function is composed of a generation loss function and a discrimination loss function; Generating loss function for: ； Discriminative loss function for: ； in, represents a matrix of all zeros, Pedestrian image samples generated by decoder G in the spatiotemporal feature encoding and decoding network , represents the concatenation of time channels, Represents dimension and consistent identity matrix; is the actual video frame sequence, the corresponding real video frame and the decoder input to the spatiotemporal feature encoding and decoding network By cascading on the time channel, we can get , is the sampling value of the real video sequence, ; The training process of the spatiotemporal consistency enhancement network uses the decoder in the spatiotemporal feature encoding and decoding network and the spatiotemporal consistency discriminator to alternately train until both reach convergence.

5. The intelligent video monitoring method based on multi-source data fusion according to claim 1, characterized in that: The feature recognition network in S4 includes a feature learning module and a feature mapping module. The feature learning module is composed of a convolutional neural network, a fully connected layer, and a batch normalization layer; the feature mapping module is composed of a random inactivation layer, a fully connected layer, and a two-norm normalization layer; the specific steps of the feature recognition network to realize feature extraction of pedestrian abnormal behavior images are as follows: The pedestrian image obtained by S3 Input feature learning module to obtain local features of the image , the specific calculation formula is as follows: ； in is the convolutional layer, is the fully connected layer, is the batch normalization layer; The local features Input feature mapping module to obtain hypersphere domain mapping features , the specific calculation formula is as follows: ； in is the random dropout layer, is a two-norm normalization layer; through the mapping of the feature mapping module, the domain detailed features of different modes are mapped to the same hyperspherical manifold to obtain the domain shared features on the hyperspherical manifold.

6. The intelligent video monitoring method based on multi-source data fusion according to claim 5, characterized in that: The feature recognition network in S4 is trained based on the metric learning strategy of hypersphere manifold and KL divergence, specifically: First, the hypersphere manifold loss function is used to train the feature recognition network to accelerate the convergence process of the model. The hypersphere manifold loss function is as follows: ； Among them, s is the proportional coefficient of the cosine value of the angle cos, which is used to accelerate the training process of the model, and C represents the number of pedestrian abnormal behavior labels. is the number of pedestrian images contained in a batch; Represents the vector angle between different behavior categories in different pedestrian images; Indicates The angle between the abnormal behavior prediction output vector of a pedestrian image and the true label vector; Then, a KL divergence constraint training feature recognition network is constructed; that is, a probability constraint method based on Kullback-Leibler discreteness is used to measure the probability prediction value of different modalities of the same person. and The similarity between them improves the modality independence of cross-modal features. arrive The KL divergence value of Calculated by the following formula: ； Since the KL divergence is asymmetric, arrive Divergence for: ； in, is the pedestrian behavior prediction result under visible light mode, Pedestrian behavior prediction results in infrared light modality.

7. The intelligent video monitoring method based on multi-source data fusion according to claim 1, characterized in that: The specific processing process of the cross-modal pedestrian feature registration network based on dual alignment feature embedding in S5 is: S51, will Input ResNet50 backbone network; the network before the last pooling layer of traditional ResNet50 is used as ResNet50 backbone network. Input the ResNet50 backbone network and output a multi-channel feature map M. The specific calculation formula is as follows: ； Where M is a three-dimensional tensor; S52, the feature map The image is randomly divided into K non-overlapping parts along the vertical direction, and then global average pooling is performed on each part to obtain fine-grained features. , for the Part , the calculation formula is as follows: ； in is a feature vector of 2048 dimensions; S53, the fine-grained features The input convolutional layer and batch normalization layer reduce The dimension is reduced to a feature vector with a dimension of 256*K; S54, respectively inputting the K feature vectors of the fine-grained features after dimensionality reduction into K fully connected layers with different weights to obtain the final pedestrian image abnormal behavior detection result.

8. The intelligent video monitoring method based on multi-source data fusion according to claim 7, characterized in that: The cross-modal pedestrian feature registration network based on dual alignment feature embedding is trained through a modality consistency metric learning strategy, including an intra-class distribution constraint loss function and an inter-class correlation constraint loss function, as follows: The intra-class distribution constraint loss function is minimized by minimizing the following formula Discreteness, obtain the intra-class distribution constraint loss function : ； in and are the means of the visible light modality pedestrian features and the infrared modality pedestrian features, and is the standard deviation of the visible light modality pedestrian features and the infrared modality pedestrian features, , , and The corresponding calculation formulas are as follows: ； ； ； ； in, Indicates The features of infrared pedestrian images, Indicates Features of pedestrian images in visible light mode; The inter-class correlation constraint loss function The specific calculation formula is as follows: ； in is the Pearson correlation coefficient matrix of a batch of infrared images The elements, is the Pearson correlation coefficient matrix of a batch of visible light images The elements, and The calculation formula is as follows: ； 。 9. An intelligent video surveillance system based on multi-source data fusion, applied to the intelligent video surveillance method according to any one of claims 1 to 8, characterized in that: include: Video acquisition module and backend server; The video acquisition module is used to acquire multi-source video data; The backend server is internally deployed with a video preprocessing module, a spatiotemporal feature encoding and decoding network, an abnormal behavior extraction module, a feature recognition network, and a cross-modal pedestrian feature registration network based on dual-aligned feature embedding; The video preprocessing module is used to standardize and remove noise from multi-source videos, and uniformly adjust spatial resolution and brightness; The spatiotemporal feature encoding and decoding module is used to input the preprocessed video data to obtain preliminary results of abnormal behavior detection of pedestrian images; An abnormal behavior extraction module is used to extract pedestrian images in abnormal behavior and obtain abnormal behavior image data; The feature recognition network is used to obtain a feature vector; The cross-modal pedestrian feature registration network based on dual-aligned feature embedding is used to obtain the final pedestrian image abnormal behavior detection result.

Images