Video real-time analysis processing method based on deep learning

Abstract

The invention relates to the field of video processing, in particular to a video real-time analysis processing method based on deep learning. The method comprises the following steps: collecting and preprocessing video data, generating a time difference chart based on an obtained video frame sequence, and extracting a feature vector; dynamically adjusting the size of a time window based on the feature vector of the time difference chart; selecting feature vectors of the time difference chart based on the size of a time window, extracting local features and global features through a nonlinear feature fusion anomaly detection method, and fusing the local features and the global features to generate an anomaly feature value; and based on the abnormal characteristic value, carrying out quantitative evaluation on the abnormal degree of the time window. The method solves the problems that a traditional video analysis method lacks adaptive capacity to dynamic changes of videos, and efficiency is low when sudden abnormal events are processed; feature extraction of video frames often depends too much on global features, and detail capture and emergency detection are insufficient. And an effective feature fusion strategy is lacked in anomaly detection, so that missing report or false report is easy to occur.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a real-time video analysis and processing method based on deep learning. Background Art

[0002] In recent years, with the development of deep learning technology, video analysis methods have been widely used in the fields of intelligent monitoring, anomaly detection and behavior recognition. Traditional video analysis methods are usually based on inter-frame differences and feature extraction, and extract static or dynamic features in the video through deep models such as convolutional neural networks to achieve understanding of video content and event recognition. At the same time, with the improvement of hardware performance, real-time analysis needs have gradually become the focus of video processing technology. Researchers are constantly exploring more efficient adaptive feature extraction and feature fusion methods to efficiently detect key events and abnormal situations in complex and changing scenarios, providing more accurate and efficient solutions for video analysis in multi-scenario applications.

[0003] However, traditional video analysis methods still face the following key problems: first, they usually use a fixed-length time window for analysis, lacking the ability to adapt to dynamic changes in the video, resulting in low efficiency in dealing with sudden abnormal events; second, feature extraction of video frames often relies too much on global features, ignoring information about local changes, which is insufficient in capturing details and detecting sudden events; in addition, in anomaly detection, there is a lack of effective feature fusion strategies, making it difficult to take into account both short-term changes and long-term trends, which easily leads to missed or false positives of abnormal events in complex scenes. Therefore, it is urgent to improve it through a new algorithm and processing technology. Summary of the invention

[0004] The present invention provides a real-time video analysis and processing method based on deep learning to solve the problems that traditional video analysis methods usually use a fixed-length time window for analysis, lack the ability to adapt to dynamic changes in the video, resulting in low efficiency in processing sudden abnormal events; feature extraction of video frames often relies too much on global features and ignores information on local changes, resulting in deficiencies in detail capture and sudden event detection; in anomaly detection, there is a lack of effective feature fusion strategies, making it difficult to take into account both short-term changes and long-term trends at the same time, which easily leads to the problem of underreporting or false reporting of abnormal events in complex scenes.

[0005] The video real-time analysis and processing method based on deep learning of the present invention specifically includes the following technical solutions:

[0006] The video real-time analysis and processing method based on deep learning includes the following steps:

[0007] S 1: Collect video data and perform preprocessing to obtain a video frame sequence; based on the video frame sequence, generate a time difference map and extract feature vectors to obtain feature vectors of the time difference map; introduce an adaptive change pattern window generation mechanism to dynamically adjust the time window size based on the feature vectors of the time difference map;

[0008] S2: Select the feature vector of the time difference map based on the time window size, extract local features and global features through nonlinear feature fusion anomaly detection method; fuse local features and global features to generate abnormal feature values; based on the abnormal feature values, quantitatively evaluate the abnormal degree of the time window and mark abnormal events in the video.

[0009] Preferably, the S1 specifically includes:

[0010] Combining the pixel value difference and gradient change between two adjacent frames in the video frame sequence, the difference attenuation control parameter and normalization adjustment parameter are introduced to calculate the time difference map.

[0011] Preferably, the S1 specifically includes:

[0012] The adaptive change pattern window generation mechanism introduces an adaptive change pattern window generation formula based on the time difference graph, and adaptively adjusts the time window size of the time difference graph.

[0013] Preferably, the S1 specifically includes:

[0014] The adaptive change pattern window generation formula introduces dynamic weights and change intensity thresholds to dynamically adjust the time window size of the time difference graph; the adaptive change pattern window generation formula is as follows:

[0015]

[0016] Among them, U i is the time window size of the time difference map of the i-th frame; i is the index variable of the frame; α is the change intensity adjustment parameter; d is the dimension of the feature vector of the time difference map; j is the index variable of the eigenvalue of the feature vector of the time difference map; V i [j] and V i+1 [j] is the jth eigenvalue of the eigenvector of the time difference graph of the i-th frame and the i+1-th frame respectively; δ j is the jth dynamic weight; β is the function adjustment parameter; ξ is the slope control parameter of the sigmoidal function; θ is the change intensity threshold.

[0017] Preferably, the S2 specifically includes:

[0018] The nonlinear feature fusion anomaly detection method introduces a multi-level feature extraction mechanism, extracts local features and global features based on the feature vectors of all time difference maps in a time window, and then combines the local features with the global features through nonlinear transformation and weighted fusion strategy to generate anomaly feature values.

[0019] Preferably, the S2 specifically includes:

[0020] By calculating the weighted square difference of the eigenvalues ​​of the eigenvector of the time difference map within the time window, combined with dynamic weights and Gaussian attenuation factors, local changes are captured and local features are obtained.

[0021] Preferably, the S2 specifically includes:

[0022] The hyperbolic tangent function and logarithmic function are introduced to perform nonlinear transformation on the feature vector of the time difference graph within the time window to obtain the global feature.

[0023] Preferably, the S2 specifically includes:

[0024] The local features and global features are nonlinearly fused to generate abnormal feature values. The calculation formula of abnormal feature values ​​is as follows:

[0025]

[0026] Among them, A i is the abnormal feature value of the time window of the time difference map of the i-th frame; i is the index variable of the frame; d is the dimension of the feature vector of the time difference map; j is the index variable of the feature value of the feature vector of the time difference map; L i [j] and G i [j] are the local eigenvalue and global eigenvalue on the jth feature dimension in the time window of the time difference map of the i-th frame, respectively; is the local feature L of the current time window i The standard deviation of the local eigenvalues ​​in ; is the global feature G of the current time window i The standard deviation of the global eigenvalues ​​in ; γ is the fusion index; ∈ is a positive number to prevent the denominator from being zero.

[0027] Preferably, the S2 specifically includes:

[0028] Based on the abnormal feature value, the abnormal degree of the time window is quantitatively evaluated; when the abnormal feature value is greater than the preset abnormal threshold, it is determined that there is an abnormal event in the current time window, the current time window is marked as an abnormal state, and the video frame sequence corresponding to the time difference map in the abnormal time window is recorded and saved.

[0029] The beneficial effects of the technical solution of the present invention are:

[0030] 1. The present invention significantly improves the detection capability of complex dynamic scenes through the generation of temporal difference maps (TDM); by combining the pixel value difference and gradient change between two adjacent frames in a video frame sequence, the temporal difference map can effectively capture the subtle changes and edge information between frames, thereby enhancing the sensitivity and robustness to dynamic events.

[0031] 2. The present invention introduces an adaptive change pattern window generation mechanism, which automatically determines the time window size by adaptively adjusting the intensity of feature changes in adjacent frames, effectively balancing the real-time and accuracy of detection. The dynamic time window can not only accurately capture sudden anomalies, but also reduce unnecessary frame processing, improve overall efficiency, and solve the lag problem of traditional fixed window methods when dealing with rapid changes or emergencies. Compared with traditional fixed window methods, it can more flexibly cope with complex environmental factors.

[0032] 3. The present invention introduces a multi-level feature extraction mechanism through the nonlinear feature fusion anomaly detection method (NFFAD), which can effectively combine local and global features to perform quantitative analysis on abnormal events; local features reflect short-term changes and enhance the ability to detect subtle changes, while global features capture long-term trend changes, improve the comprehensive analysis capability of complex scenarios, reduce false alarm rate and missed alarm rate, and make anomaly detection more accurate and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the deep learning-based real-time video analysis and processing method described in the present invention. DETAILED DESCRIPTION

[0034] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0036] The specific scheme of the real-time video analysis and processing method based on deep learning provided by the present invention is described in detail below with reference to the accompanying drawings.

[0037] Refer to the attached Figure 1, which shows a flow chart of a method for real-time video analysis and processing based on deep learning provided by an embodiment of the present invention, the method comprising the following steps:

[0038] S 1: Collect video data and perform preprocessing to obtain a video frame sequence; based on the video frame sequence, generate a time difference map and extract feature vectors to obtain feature vectors of the time difference map; introduce an adaptive change pattern window generation mechanism to dynamically adjust the time window size based on the feature vectors of the time difference map;

[0039] First, data collection and preprocessing are performed to obtain a video frame sequence. In the data collection stage, video data is obtained by acquiring a video stream in real time through a camera device, or by importing a recorded video file from a storage device. Video data is stored in the form of continuous video frames, and each frame contains image information with a fixed resolution. The image information with a fixed resolution determines the number of pixels in each frame. The pixel value is the basic unit of image information in a video frame. The inter-frame dynamics are captured and abnormal conditions in the video are identified by detecting changes in pixel values ​​in real time. In the preprocessing stage, the video frames are preliminarily processed, including: frame rate standardization, image scaling, denoising and illumination equalization operations to ensure the consistency and accuracy of subsequent analysis. The preliminarily processed video frames are subjected to key frame extraction, and redundant frames are removed to obtain a video frame sequence to ensure the continuity and representativeness of the video frame sequence. The preprocessing steps are well known to those skilled in the art and will not be elaborated here.

[0040] Then, based on the video frame sequence F = {F1, F2, ..., F t}, F i is the i-th video frame, i ranges from 1 to t, t represents the length of the video frame sequence, and the temporal difference map (TDM) is introduced to reflect the dynamic information of the video frame. Specifically, based on the difference and gradient change of the pixel values ​​between two adjacent frames in the video frame sequence, the temporal difference map is calculated and generated; based on the temporal difference map of adjacent frames in the video frame sequence, the dynamic change information of the video frame is extracted to reflect the local motion or change characteristics between the current frame and the adjacent frames, that is, the inter-frame characteristics. For the last video frame, the temporal difference map is not calculated. The temporal difference map calculation formula is as follows:

[0041]

[0042] Among them, TDM i (x, y) is an element in the time difference map of the i-th frame, indicating that the i-th frame F at the coordinate (x, y) i and the i+1th frame F i+1 The difference and gradient change of pixel values ​​between the two frames; i is the frame index variable used to identify a specific frame in the video frame sequence; (x, y) is the two-dimensional coordinate of the pixel, where x is the row coordinate and y is the column coordinate; Fi (x, y) and F i+1 (x, y) represents the pixel value at the coordinate (x, y) of the i-th frame and the i+1-th frame respectively; λ is the difference attenuation control parameter, which is used to adjust the influence of inter-frame gradient changes on the time difference map and is obtained through experimental tuning; and are the gradient values ​​of the i-th frame and the i+1-th frame at the coordinates (x, y), which are calculated by a convolution operator (such as a Sobel operator), reflecting the direction and magnitude of the image intensity change in the video frame, representing the edge information of the image in the video frame, and used to enhance the feature expression ability of the time difference map. The convolution operator is a prior art and will not be described in detail here; η is a normalization adjustment parameter, which is an adjustment factor for the normalized gradient sum, used to control the contribution of the overall gradient intensity in the time difference map, and is obtained by experimental tuning;

[0043] The calculation formula of the temporal difference map (TDM) captures the dynamic changes between video frames by combining the pixel value difference and gradient change between two adjacent frames in the video frame sequence. By introducing the difference attenuation control parameter and the normalization adjustment parameter, the sensitivity to the edge changes of the image in the video frame is enhanced, the influence of the high gradient area is suppressed, and the ability of the temporal difference map to express subtle changes in complex scenes is improved;

[0044] Furthermore, a convolutional neural network is used to extract features from the time difference graph to obtain a feature vector of the time difference graph. First, features are extracted through multi-layer convolution operations to capture the change patterns of different scales; second, after each convolution layer, an activation function is applied to introduce nonlinear transformations to enhance the ability to fit complex data; next, downsampling is performed through a pooling layer, such as using maximum pooling or average pooling; finally, the extracted features are flattened and processed through a fully connected layer to generate a final high-dimensional feature vector. The convolutional neural network is a prior art and will not be described in detail here;

[0045] Next, an adaptive change pattern window generation mechanism is introduced. By calculating the feature change intensity between the feature vectors of the time difference graph, the range of the time window when analyzing the time difference graph of the i-th frame is adaptively adjusted to improve the detection sensitivity and real-time performance of abnormal events. The adaptive change pattern window generation formula is as follows:

[0046]

[0047] Among them, U i is the time window size of the time difference graph of the i-th frame, which is the range of the time window when analyzing the time difference graph of the i-th frame; i is the index variable of the frame; is the feature change intensity between the feature vectors of the time difference graph; α is the change intensity adjustment parameter, which is used to control the influence of the feature change intensity between the feature vectors of the time difference graph on the time window size of the time difference graph, and is obtained through experimental tuning; d is the dimension of the feature vector of the time difference graph; j is the index variable of the eigenvalue of the feature vector of the time difference graph; V i [j] and V i+1 [j] is the jth eigenvalue of the eigenvector of the time difference graph of the i-th frame and the i+1-th frame respectively; δ j is the jth dynamic weight, which is used to control the influence of the eigenvalue and ensure that the generation of the time window of the time difference map is more stable. It is set according to the specific implementation scenario. β is the function adjustment parameter, which is used to control the influence of the S-shaped function (sigmoidal function) on the time window size of the time difference map. It is obtained through experimental tuning. ξ is the slope control parameter of the sigmoidal function, which is used to adjust the response rate of the feature change intensity between the eigenvectors of the time difference map. It is obtained through experimental tuning. θ is the change intensity threshold, which is set according to the specific implementation scenario and is used to distinguish normal changes from significant changes between video frames. When |V i [j]-V i+1 When the value of [j]| exceeds the change intensity threshold, the value of the sigmoidal function increases rapidly;

[0048] The adaptive change pattern window generation formula dynamically adjusts the time window size based on the feature vector of the time difference graph. Combining logarithmic function and S-shaped function, it captures the overall trend and strengthens the response to significant changes; introduces dynamic weights and change intensity thresholds to ensure that the time window remains stable when adapting to changes, flexibly adjusts the analysis scope, and improves the accuracy and efficiency of anomaly detection.

[0049] S2: Select the feature vector of the time difference map based on the time window size, extract local features and global features through nonlinear feature fusion anomaly detection method; fuse local features and global features to generate abnormal feature values; based on the abnormal feature values, quantitatively evaluate the abnormal degree of the time window and mark abnormal events in the video.

[0050] Based on the time window size, the feature vector of the time difference map is selected; specifically, starting from the feature vector of the current time difference map, feature vectors are continuously selected according to the time window size, and the feature vectors selected in the time window are renumbered to facilitate consistency analysis and processing within the time window range; the expression of the time window of the i-th frame time difference map is

[0051] Through the nonlinear feature fusion anomaly detection method (NFFAD), a multi-level feature extraction mechanism is introduced to extract local features and global features, and then perform fusion analysis. The nonlinear feature fusion anomaly detection method first extracts local features and global features based on all feature vectors in the time window, and then combines local features with global features through nonlinear transformation and weighted fusion strategy to generate abnormal feature values, quantitatively analyze the degree of abnormality in the current time window, and provide a basis for identifying sudden abnormal events.

[0052] Local features are used to capture short-term changes in feature vectors within a time window. Each time window generates a local feature, which is represented in the form of a vector and is consistent with the feature vector dimension of the time difference graph. The larger the local feature value, the greater the possibility of large or frequent changes in the current time window, and the greater the possibility of abnormal events or sudden behaviors. The local feature extraction formula is as follows:

[0053]

[0054] Among them, L i [j] is the local eigenvalue on the jth feature dimension in the time window of the time difference map of the i-th frame; i is the index variable of the frame; j is the index variable of the eigenvalue of the eigenvector of the time difference map, indicating the jth feature dimension; U i is the time window size of the time difference map of the i-th frame; k and n are the indices of the feature vector in the current time window; V k [j] and V n [j] is the j-th eigenvalue of the k-th and n-th eigenvectors in the current time window, respectively; exp is the exponential operation; is the standard deviation of Gaussian decay, which is used to control the speed of weight decay and is set using expert experience; is the Gaussian decay factor, which gives higher weights to the eigenvectors close to the first eigenvector in the current time window; w k is the dynamic weight of the kth feature vector in the current time window, Calculated, where V k-1 [j], V k+1 [j], V n-1 [j] and V n+1 [j] is the jth eigenvalue of the k-1th, k+1th, n-1th, and n+1th eigenvectors in the current time window, respectively; ∈ is a positive number to prevent the denominator from being zero, which is set according to the specific implementation scenario. i When , the feature vectors corresponding to the current k and n are used to replace the missing adjacent feature vectors.

[0055] The local feature extraction formula combines dynamic weights and Gaussian attenuation factors to achieve detailed capture of local features. By calculating the weighted square difference of the eigenvalues ​​and introducing the Gaussian attenuation factor, it captures local changes and adjusts the contribution of different eigenvectors within the time window, thereby enhancing the sensitivity to local changes.

[0056] Global features are used to capture the overall change trend within a time window, and each time window generates a global feature. Global features are represented in the form of vectors and are consistent with the feature vector dimension of the time difference graph. The larger the global feature value, the more likely there is a continuous change process in the entire time window, and the greater the possibility of long-term accumulated abnormal behavior. The global feature extraction formula is as follows:

[0057]

[0058] Among them, G i [j] is the global eigenvalue on the jth feature dimension in the time window of the time difference map of the i-th frame; tanh is the hyperbolic tangent function; σ j and μ j are the standard deviation and mean of the j-th eigenvalue of all eigenvectors in the current time window, respectively.

[0059] The global feature formula enhances the capture of overall trends through nonlinear processing, uses the hyperbolic tangent function to smooth extreme values ​​to prevent excessive extreme values ​​from affecting the final results, and uses the logarithmic function to amplify feature deviations and increase attention to significant changes (feature values ​​that deviate from the mean). It can effectively identify continuous changes within the time window and improve the accuracy of identifying anomalies in complex scenarios.

[0060] The local features and global features are nonlinearly fused to generate an abnormal feature value; the abnormal feature value is expressed in the form of a scalar and is used to quantify the degree of abnormality in the current time window. Each time window generates an abnormal feature value. The larger the abnormal feature value, the greater the possibility of abnormal behavior in the frame sequence corresponding to the time window. The abnormal feature value calculation formula is as follows:

[0061]

[0062] Among them, A i is the abnormal characteristic value of the time window of the time difference map of the i-th frame; L i [j] and G i [j] are the local eigenvalue and global eigenvalue on the jth feature dimension in the time window of the time difference map of the i-th frame, respectively; is the local feature L of the current time window i The standard deviation of the local eigenvalues ​​in ; is the global feature G of the current time windowi The standard deviation of the global eigenvalues ​​in ; γ is the fusion index, which is used to control the degree of nonlinearity when combining local features with global features and is obtained through experimental tuning.

[0063] Finally, the calculated abnormal feature value is used to quantitatively evaluate the degree of abnormality in the time window. The abnormal threshold is set according to the specific implementation scenario to detect and mark abnormal events in the video in real time;

[0064] Specifically, when the abnormal feature value is greater than the preset abnormal threshold, it is determined that there is an abnormal event in the current time window. At this time, the current time window is marked as an abnormal state, the corresponding processing flow is triggered, an alarm is issued, and the video frame sequence corresponding to the time difference graph in the abnormal time window is recorded, and the video frame sequence is saved to the database for subsequent analysis; at the same time, the timestamp and position of the abnormal event are automatically generated for rapid positioning and backtracking, thereby improving the real-time and processing efficiency of video surveillance and ensuring that abnormal events can be detected and responded to in a timely manner.

[0065] In summary, the real-time video analysis and processing method based on deep learning is completed.

[0066] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0067] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A video real-time analysis and processing method based on deep learning, characterized in that: The following steps are involved: S1: Collect video data and perform preprocessing to obtain a video frame sequence; based on the video frame sequence, generate a time difference map and extract feature vectors to obtain feature vectors of the time difference map; introduce an adaptive change pattern window generation mechanism to dynamically adjust the time window size based on the feature vectors of the time difference map; S2: Select the feature vector of the time difference map based on the time window size, extract local features and global features through nonlinear feature fusion anomaly detection method; fuse local features and global features to generate abnormal feature values; based on the abnormal feature values, quantitatively evaluate the abnormal degree of the time window and mark abnormal events in the video.

2. The method for real-time video analysis and processing based on deep learning according to claim 1, characterized in that: The S1 specifically includes: Combining the pixel value difference and gradient change between two adjacent frames in the video frame sequence, the difference attenuation control parameter and normalization adjustment parameter are introduced to calculate the time difference map.

3. The video real-time analysis and processing method based on deep learning according to claim 2 is characterized in that: The S1 specifically includes: The adaptive change pattern window generation mechanism introduces an adaptive change pattern window generation formula based on the time difference graph, and adaptively adjusts the time window size of the time difference graph.

4. The method for real-time video analysis and processing based on deep learning according to claim 3 is characterized in that: The S1 specifically includes: The adaptive change pattern window generation formula introduces dynamic weights and change intensity thresholds to dynamically adjust the time window size of the time difference graph; the adaptive change pattern window generation formula is as follows: Among them, U i is the time window size of the time difference map of the i-th frame; i is the index variable of the frame; α is the change intensity adjustment parameter; d is the dimension of the feature vector of the time difference map; j is the index variable of the eigenvalue of the feature vector of the time difference map; V i [j] and V i+1 [j] is the jth eigenvalue of the eigenvector of the time difference graph of the i-th frame and the i+1-th frame respectively; δ j is the jth dynamic weight; β is the function adjustment parameter; ξ is the slope control parameter of the sigmoidal function; θ is the change intensity threshold.

5. The method for real-time video analysis and processing based on deep learning according to claim 1, characterized in that: The S2 specifically includes: The nonlinear feature fusion anomaly detection method introduces a multi-level feature extraction mechanism, extracts local features and global features based on the feature vectors of all time difference maps in a time window, and then combines the local features with the global features through nonlinear transformation and weighted fusion strategy to generate anomaly feature values.

6. The method for real-time video analysis and processing based on deep learning according to claim 5 is characterized in that: The S2 specifically includes: By calculating the weighted square difference of the eigenvalues ​​of the eigenvector of the time difference map within the time window, combined with dynamic weights and Gaussian attenuation factors, local changes are captured and local features are obtained.

7. The method for real-time video analysis and processing based on deep learning according to claim 6 is characterized in that: The S2 specifically includes: The hyperbolic tangent function and logarithmic function are introduced to perform nonlinear transformation on the feature vector of the time difference graph within the time window to obtain the global feature.

8. The method for real-time video analysis and processing based on deep learning according to claim 7 is characterized in that: The S2 specifically includes: The local features and global features are nonlinearly fused to generate abnormal feature values. The calculation formula of abnormal feature values ​​is as follows: Among them, A i is the abnormal feature value of the time window of the time difference map of the i-th frame; i is the index variable of the frame; d is the dimension of the feature vector of the time difference map; j is the index variable of the feature value of the feature vector of the time difference map; L i [j] and G i [j] are the local eigenvalue and global eigenvalue on the jth feature dimension in the time window of the time difference map of the i-th frame, respectively; is the local feature L of the current time window i The standard deviation of the local eigenvalues ​​in ; is the global feature G of the current time window i The standard deviation of the global eigenvalues ​​in ; γ is the fusion index; ∈ is a positive number to prevent the denominator from being zero.

9. The method for real-time video analysis and processing based on deep learning according to claim 8, characterized in that: The S2 specifically includes: Based on the abnormal feature value, the abnormal degree of the time window is quantitatively evaluated; when the abnormal feature value is greater than the preset abnormal threshold, it is determined that there is an abnormal event in the current time window, the current time window is marked as an abnormal state, and the video frame sequence corresponding to the time difference map in the abnormal time window is recorded and saved.

Images