A video surveillance early warning method and system based on multimodal behavior patterns

Abstract

The present invention provides a video surveillance and early warning method based on multimodal behavior patterns, comprising the following steps: extracting multimodal data according to a video stream, performing preprocessing, and aligning and fusing the preprocessed multimodal data, wherein the multimodal data includes RGB frames, optical flow fields, key points of human skeletons, and scene semantic segmentation maps; performing short-term behavior pattern characterization on the fused multimodal data, establishing a behavior pattern library and constructing a scene behavior baseline; updating the dynamic behavior pattern library based on online learning, reducing the weight of historical data in combination with a time attenuation factor, and adjusting the abnormality judgment threshold according to the complexity of the current scene; performing multi-level early warning according to the abnormality judgment threshold, and feedback optimization; the method of the present invention provides an intelligent video surveillance and early warning solution that can automatically learn scene behavior patterns, dynamically adjust the early warning threshold, and fuse multi-dimensional feature analysis, thereby solving the problems of high false alarm rate and poor adaptability of the existing technology.

Description

Technical Field

[0001] The present invention relates to the field of intelligent video surveillance technology, and in particular to a multimodal behavior pattern-based video surveillance early warning method and system. Background Art

[0002] Current video surveillance systems are mostly based on static rules (such as area intrusion and object left behind detection) or single behavior recognition (such as running and falling), and have the following defects: high false alarm rate: lighting changes, occlusions, and complex backgrounds can easily trigger false alarms; poor adaptability: unable to dynamically learn behavioral patterns in scenes, and difficult to cope with changes in crowd density or new behavior types; lack of correlation analysis: isolated analysis of single frames or short clips, ignoring the temporal and spatial continuity of behavior. Summary of the Invention

[0003] The main purpose of the present invention is to overcome the above-mentioned defects in the prior art and provide a video surveillance and early warning method and system based on multimodal behavior patterns. The method provides an intelligent video surveillance and early warning solution that can automatically learn scene behavior patterns, dynamically adjust warning thresholds, and integrate multi-dimensional feature analysis, thereby solving the problems of high false alarm rate and poor adaptability of the prior art.

[0004] The present invention adopts the following technical solutions:

[0005] A multi-modal behavior pattern video surveillance early warning method includes the following steps:

[0006] Extracting multimodal data from the video stream, preprocessing it, and aligning and fusing the preprocessed multimodal data, wherein the multimodal data includes RGB frames, optical flow fields, human skeleton key points, and scene semantic segmentation maps;

[0007] Characterize short-term behavior patterns of the fused multimodal data, establish a behavior pattern library, and construct a scenario behavior baseline;

[0008] Update the dynamic behavior pattern library based on online learning, reduce the weight of historical data by combining the time decay factor, and adjust the anomaly judgment threshold according to the complexity of the current scenario;

[0009] Multi-level warnings are issued based on abnormality judgment thresholds, and feedback is provided for optimization.

[0010] Specifically, the alignment and fusion of the pre-processed multimodal data includes:

[0011] Strictly align multimodal data through hardware timestamps;

[0012] For low frame rate modes, linear interpolation is used to fill in the intermediate frames;

[0013] The optical flow field is superimposed on the RGB frame channel to obtain the first fusion data, and the human skeleton key point trajectory and semantic segmentation map results are weighted through the attention mechanism to obtain the second fusion data.

[0014] Specifically, the fused multimodal data is characterized by short-term behavior patterns, a behavior pattern library is established, and a scenario behavior baseline is constructed; specifically:

[0015] The 3DCNN architecture is used to extract the spatiotemporal features of the first fused data and the second fused data, that is, to construct a behavior pattern library for short-term behavior pattern features;

[0016] The short-term behavior pattern features of the behavior pattern library are spliced ​​according to the time window, with a total span of M vectors, which are input into the bidirectional LSTM and output the scene behavior baseline vector.

[0017] Specifically, the behavior pattern library is dynamically updated based on online learning, and the weight of historical data is reduced by combining the time decay factor. Specifically:

[0018] Acquire short-term behavior pattern features in real time and perform sliding window incremental updates on the behavior pattern library;

[0019] The historical data in the behavior pattern library is weighted according to the generation time. The specific weight settings are as follows:

[0020] ;

[0021] in, is the initial weight, which is 1, is the decay rate, ranging from 0.001 to 0.01; The interval between the current time and the data generation time.

[0022] Specifically, the anomaly determination threshold is adjusted according to the complexity of the current scenario, specifically:

[0023] Extract the parameters of the current scene, including crowd density, light intensity, and motion activity, and perform normalization to obtain the normalized crowd density , light intensity ;

[0024] Dynamic anomaly determination threshold :

[0025] ;

[0026] in, is the basic threshold, is the adjustment coefficient, and the higher the crowd density and the stronger the light intensity, the larger the dynamic anomaly judgment threshold.

[0027] Specifically, multi-level early warnings are carried out according to the anomaly determination threshold and feedback optimization is performed, specifically as follows:

[0028] Obtain the similarity score S(t) between the current behavior characteristics and the baseline;

[0029] If S(t) < T(t): Determine it as an anomaly and enter the early warning;

[0030] If S(t) ≥ T(t) but the administrator marks it as an anomaly: Perform feedback optimization and trigger incremental learning of the model.

[0031] On the other hand, the present invention also provides a multi-modal behavior pattern video surveillance early warning system, including:

[0032] Data processing unit: Extract multi-modal data according to the video stream, perform preprocessing, and align and fuse the preprocessed multi-modal data. The multi-modal data includes RGB frames, optical flow fields, human body skeleton key points, and scene semantic segmentation maps;

[0033] Behavior baseline construction unit: Perform short-term behavior pattern features on the fused multi-modal data, and establish a behavior pattern library and a scene behavior baseline construction;

[0034] Update and adjustment unit: Update the dynamic behavior pattern library based on online learning, reduce the weight of historical data in combination with the time decay factor, and adjust the anomaly determination threshold according to the current scene complexity;

[0035] Early warning unit: Perform multi-level early warnings according to the anomaly determination threshold and feedback optimization.

[0036] On yet another aspect, the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. Wherein, when the processor executes the computer program, it implements the steps of a multi-modal behavior pattern video surveillance early warning method.

[0037] On yet another aspect, a computer-readable storage medium is characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of a multi-modal behavior pattern video surveillance early warning method.

[0038] From the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention proposes a video surveillance and early warning method based on multimodal behavior patterns, comprising the following steps: extracting multimodal data according to a video stream, performing preprocessing, and aligning and fusing the preprocessed multimodal data, wherein the multimodal data includes RGB frames, optical flow fields, key points of human skeletons, and scene semantic segmentation maps; performing short-term behavior pattern characterization on the fused multimodal data, establishing a behavior pattern library and a scene behavior baseline; updating the dynamic behavior pattern library based on online learning, reducing the weight of historical data in combination with a time attenuation factor, and adjusting the abnormality judgment threshold according to the complexity of the current scene; performing multi-level early warning according to the abnormality judgment threshold, and feedback optimization; the method of the present invention provides an intelligent video surveillance and early warning solution that can automatically learn scene behavior patterns, dynamically adjust the warning threshold, and integrate multi-dimensional feature analysis, thereby solving the problems of high false alarm rate and poor adaptability of the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a multimodal behavior pattern video surveillance early warning method provided by an embodiment of the present invention;

[0041] FIG2 is a structural diagram of a multimodal behavior pattern-based video surveillance and early warning system provided by an embodiment of the present invention;

[0042] FIG3 is a schematic diagram of an embodiment of an electronic device provided by an embodiment of the present invention;

[0043] FIG4 is a schematic diagram of an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described below through specific embodiments.

[0045] The present invention proposes a video surveillance and early warning method based on multimodal behavior patterns. The method provides an intelligent video surveillance and early warning solution that can automatically learn scene behavior patterns, dynamically adjust early warning thresholds, and integrate multi-dimensional feature analysis, thereby solving the problems of high false alarm rate and poor adaptability of existing technologies.

[0046] like Figure 1 The present invention provides a multi-modal behavior pattern video surveillance early warning method, which specifically includes the following steps:

[0047] S101: extracting multimodal data from the video stream, performing preprocessing, and aligning and fusing the preprocessed multimodal data, wherein the multimodal data includes RGB frames, optical flow fields, human skeleton key points, and scene semantic segmentation maps;

[0048] By integrating multi-dimensional visual information with dynamic environmental parameters, this technology addresses the high false detection rate and poor environmental adaptability inherent in traditional video surveillance, which are often caused by a single data source. This is achieved as follows: In this embodiment, real-time video streaming (supporting visible light / infrared dual-mode input, resolution ≥ 1080P, frame rate ≥ 25fps) is implemented.

[0049] Extract modal:

[0050] RGB frames are used to obtain raw color information for object detection and scene semantic analysis. Extraction method: Intercept the video stream frame by frame, preserving timestamp alignment.

[0051] Optical flow quantifies pixel-level motion vectors to capture short-term behavioral dynamics (such as rapid movements and body swaying). Algorithm of choice: Farneback dense optical flow, which calculates full-image pixel motion and is suitable for complex scenes. Output format: 2D vector field (horizontal and vertical displacement matrices) with the same resolution as the original frame.

[0052] Human skeleton keypoints are used to identify 18 key points on the human body (head, shoulders, elbows, wrists, etc.) and construct spatiotemporal trajectories of poses. Multi-target tracking is integrated with the DeepSORT algorithm to address ID jumps in occluded scenes. Lightweight deployment: MobileNetV3 is used as the backbone network of OpenPose, enabling real-time inference at the edge (<30ms / frame).

[0053] The purpose of the scene semantic segmentation map (Mask R-CNN) is to segment static elements (such as walls and railings) from dynamic objects (such as packages and vehicles) in the scene, assisting in behavioral context analysis.

[0054] Data preprocessing, preprocessing goal: eliminate environmental interference and improve the robustness of subsequent feature extraction;

[0055] The illumination normalization CLAHE algorithm is used to solve the problem of target detection failure caused by sudden illumination changes (such as backlighting and shadows). The RGB frame is converted to the LAB color space, and only the L channel (brightness) is subjected to contrast-limited adaptive histogram equalization.

[0056] The ViBe background modeling algorithm is used to solve the problem of static objects (such as fixed facilities) being misjudged as foreground targets. Specifically, it includes dynamic background update: each pixel maintains a background model containing 20 samples, randomly replacing historical samples (with a replacement probability of 1 / 16); foreground detection: if the difference between the current pixel value and at least two samples in the background model is less than a threshold (set to 20), it is judged as background.

[0057] Specifically, the alignment and fusion of the pre-processed multimodal data includes:

[0058] Strictly align multimodal data through hardware timestamps;

[0059] For low frame rate modes, linear interpolation is used to fill in the intermediate frames;

[0060] The optical flow field is superimposed on the RGB frame channel to form a 5-channel input to obtain the first fused data (R / G / B / Flow_X / Flow_Y) for 3D CNN processing, and the human skeleton key point trajectory and semantic segmentation map results are weighted through the attention mechanism to obtain the second fused data.

[0061] formula:

[0062] ;

[0063] Among them, σ is the Sigmoid function, W1 and W2 are learnable parameters.

[0064] S102: Characterize the short-term behavior patterns of the fused multimodal data, establish a behavior pattern library, and construct a scenario behavior baseline;

[0065] Specifically, the fused multimodal data is characterized by short-term behavior patterns, a behavior pattern library is established, and a scenario behavior baseline is constructed; specifically:

[0066] The 3DCNN architecture is used to extract the spatiotemporal features of the first and second fused data, such as sudden acceleration and physical conflict, which are short-term behavior pattern features, and a behavior pattern library is constructed.

[0067] The first fused data and the second fused data have a time span of 5 seconds and a 3D CNN (C3D network) architecture. The 3D CNN model is trained first. The training data includes:

[0068] Short-term behavior classification: Label definition: 20 basic behaviors are predefined (such as "walking", "running", "raising hands", and "throwing").

[0069] Training strategy: Pre-training: Transfer learning based on the Kinetics-400 dataset (400,000 video clips).

[0070] Fine-tuning: Optimize the classification boundaries using target scene data (e.g., bank surveillance videos).

[0071] The short-term behavior pattern features of the behavior pattern library are spliced ​​according to the time window, with a total span of M vectors, which are input into the bidirectional LSTM, and the scene behavior baseline vector is output to establish a baseline to distinguish "statistical anomalies" from "real threats."

[0072] In this example, short-term behavioral pattern features (4096 dimensions) are concatenated by time window (one vector every 5 seconds), with a total span of 30 minutes (360 vectors in total). A bidirectional LSTM is stacked in two layers, with 256 hidden units in each layer, to capture temporal dependencies.

[0073] S103: Update the dynamic behavior pattern library based on online learning, reduce the weight of historical data by combining the time decay factor, and adjust the anomaly judgment threshold according to the complexity of the current scenario;

[0074] Specifically, the behavior pattern library is dynamically updated based on online learning, and the weight of historical data is reduced by combining the time decay factor. Specifically:

[0075] Acquire short-term behavior pattern features in real time and perform sliding window incremental updates on the behavior pattern library;

[0076] The historical data in the behavior pattern library is weighted according to the generation time. The specific weight settings are as follows:

[0077] ;

[0078] in, is the initial weight, which is 1, is the decay rate, ranging from 0.001 to 0.01; The interval between the current time and the data generation time.

[0079] Specifically, the anomaly determination threshold is adjusted according to the complexity of the current scenario, specifically:

[0080] Extract the parameters of the current scene, including crowd density, light intensity, and motion activity, and perform normalization to obtain the normalized crowd density , light intensity ;

[0081] Crowd density D(t): Count the number of dynamic targets / area using the Mask R-CNN segmentation results.

[0082] Light intensity L(t): The mean value of the luminance channel (LAB color space L value) is extracted from the RGB frame.

[0083] Dynamic anomaly determination threshold :

[0084] ;

[0085] in, is the basic threshold, is the adjustment coefficient, and the higher the crowd density and the stronger the light intensity, the larger the dynamic anomaly judgment threshold.

[0086] S104: Perform multi-level early warnings according to the anomaly determination threshold and provide feedback for optimization.

[0087] Specifically, performing multi-level early warnings according to the anomaly determination threshold and providing feedback for optimization is as follows:

[0088] Obtain the similarity score S(t) between the current behavior characteristics and the baseline; the similarity is calculated as the cosine similarity between the current behavior characteristics and the baseline.

[0089] If S(t) < T(t): Determine it as an anomaly and enter the early warning.

[0090] If S(t) ≥ T(t) but the administrator marks it as an anomaly: Provide feedback for optimization and trigger incremental learning of the model.

[0091] As shown in Figure 2, an embodiment of the present invention also provides a multi-modal behavior pattern video surveillance early warning system, including:

[0092] Data processing unit 201: Extract multi-modal data from the video stream, perform preprocessing, and align and fuse the preprocessed multi-modal data. The multi-modal data includes RGB frames, optical flow fields, human body skeleton key points, and scene semantic segmentation maps.

[0093] By fusing multi-dimensional visual information and dynamic environmental parameters, problems such as high false detection rates and poor environmental adaptability caused by single data sources in traditional video surveillance are solved. The specific implementation is as follows: In the embodiment, a real-time video stream (supporting visible light / infrared dual-mode input, resolution ≥ 1080P, frame rate ≥ 25fps).

[0094] Extracted modalities:

[0095] RGB frames, the purpose is to obtain the original color information for object detection and scene semantic analysis. Extraction method: Intercept the video stream frame by frame and retain the timestamp alignment.

[0096] Optical flow fields, the purpose is to quantify pixel-level motion vectors and capture short-term behavior dynamics (such as fast movement, limb swinging). Algorithm selection: Farneback dense optical flow: Calculate the motion of all image pixels, suitable for complex scenes. Output format: Two-dimensional vector field (horizontal / vertical displacement matrix), with the same resolution as the original frame.

[0097] Human body skeleton key points, the purpose is to identify 18 key points of the human body (head, shoulders, elbows, wrists, etc.) and construct a pose spatio-temporal trajectory. Use multi-object tracking: Integrate the DeepSORT algorithm to solve the ID jump problem in occlusion scenarios. Lightweight deployment: Use MobileNetV3 as the backbone network of OpenPose to achieve real-time inference at the edge (<30ms / frame).

[0098] The purpose of the scene semantic segmentation map (Mask R-CNN) is to segment static elements (such as walls and railings) from dynamic objects (such as packages and vehicles) in the scene, assisting in behavioral context analysis.

[0099] Data preprocessing, preprocessing goal: eliminate environmental interference and improve the robustness of subsequent feature extraction;

[0100] The illumination normalization CLAHE algorithm is used to solve the problem of target detection failure caused by sudden illumination changes (such as backlighting and shadows). The RGB frame is converted to the LAB color space, and only the L channel (brightness) is subjected to contrast-limited adaptive histogram equalization.

[0101] The ViBe background modeling algorithm is used to solve the problem of static objects (such as fixed facilities) being misjudged as foreground targets. Specifically, it includes dynamic background update: each pixel maintains a background model containing 20 samples, randomly replacing historical samples (with a replacement probability of 1 / 16); foreground detection: if the difference between the current pixel value and at least two samples in the background model is less than a threshold (set to 20), it is judged as background.

[0102] Specifically, the alignment and fusion of the pre-processed multimodal data includes:

[0103] Strictly align multimodal data through hardware timestamps;

[0104] For low frame rate modes, linear interpolation is used to fill in the intermediate frames;

[0105] The optical flow field is superimposed on the RGB frame channel to form a 5-channel input to obtain the first fused data (R / G / B / Flow_X / Flow_Y) for 3D CNN processing, and the human skeleton key point trajectory and semantic segmentation map results are weighted through the attention mechanism to obtain the second fused data.

[0106] formula:

[0107] ;

[0108] Among them, σ is the Sigmoid function, W1 and W2 are learnable parameters.

[0109] Behavior baseline construction unit 202: performs short-term behavior pattern characterization on the fused multimodal data, establishes a behavior pattern library and constructs a scenario behavior baseline;

[0110] Specifically, the fused multimodal data is characterized by short-term behavior patterns, a behavior pattern library is established, and a scenario behavior baseline is constructed; specifically:

[0111] The 3DCNN architecture is used to extract the spatiotemporal features of the first and second fused data, such as sudden acceleration and physical conflict, which are short-term behavior pattern features, and a behavior pattern library is constructed;

[0112] The first fused data and the second fused data have a time span of 5 seconds and a 3D CNN (C3D network) architecture. The 3D CNN model is trained first. The training data includes:

[0113] Short-term behavior classification: Label definition: 20 basic behaviors are predefined (such as "walking", "running", "raising hands", and "throwing").

[0114] Training strategy: Pre-training: Transfer learning based on the Kinetics-400 dataset (400,000 video clips).

[0115] Fine-tuning: Optimize the classification boundaries using target scene data (e.g., bank surveillance videos).

[0116] The short-term behavior pattern features of the behavior pattern library are spliced ​​according to the time window, with a total span of M vectors, which are input into the bidirectional LSTM, and the scene behavior baseline vector is output to establish a baseline to distinguish "statistical anomalies" from "real threats."

[0117] In this example, short-term behavioral pattern features (4096 dimensions) are concatenated by time window (one vector every 5 seconds), with a total span of 30 minutes (360 vectors in total). A bidirectional LSTM is stacked in two layers, with 256 hidden units in each layer, to capture temporal dependencies.

[0118] Update adjustment unit 203: updates the dynamic behavior pattern library based on online learning, reduces the weight of historical data in combination with the time decay factor, and adjusts the anomaly determination threshold according to the complexity of the current scenario;

[0119] Specifically, the behavior pattern library is dynamically updated based on online learning, and the weight of historical data is reduced by combining the time decay factor. Specifically:

[0120] Acquire short-term behavior pattern features in real time and perform sliding window incremental updates on the behavior pattern library;

[0121] The historical data in the behavior pattern library is weighted according to the generation time. The specific weight settings are as follows:

[0122] ;

[0123] in, is the initial weight, which is 1, is the decay rate, ranging from 0.001 to 0.01; The interval between the current time and the data generation time.

[0124] Specifically, the abnormal determination threshold is adjusted according to the current scene complexity, specifically as follows:

[0125] Extract the parameters in the current scene, including crowd density, light intensity, and movement activity, and perform normalization to obtain the normalized crowd density and light intensity ;

[0126] Crowd density D(t): Count the number of dynamic objects / area of the region through the segmentation result of Mask R-CNN.

[0127] Light intensity L(t): Extract the mean value of the luminance channel (L value in the LAB color space) from the RGB frame.

[0128] Dynamic abnormal determination threshold :

[0129] ;

[0130] Among them, is the basic threshold, is the adjustment coefficient, and the higher the crowd density, the stronger the light intensity, and the larger the dynamic abnormal determination threshold.

[0131] Early warning unit 204: Perform multi-level early warning according to the abnormal determination threshold and provide feedback for optimization.

[0132] Specifically, perform multi-level early warning according to the abnormal determination threshold and provide feedback for optimization, specifically as follows:

[0133] Obtain the similarity score S(t) between the current behavior feature and the baseline; the similarity is calculated as the cosine similarity between the current behavior feature and the baseline;

[0134] If S(t) < T(t): Determine it as abnormal and enter the early warning;

[0135] If S(t) ≥ T(t) but the administrator marks it as abnormal: Provide feedback for optimization and trigger incremental learning of the model.

[0136] As shown in Figure 3, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored on the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it implements a multi-modal behavior pattern video surveillance early warning method provided by an embodiment of the present invention.

[0137] In the specific implementation process, when the processor 320 executes the computer program 311, it can implement Figure 1 any implementation manner in the corresponding embodiment.

[0138] Since the electronic device introduced in this embodiment is a device used to implement a data processing device in the embodiment of the present invention, based on the method introduced in the embodiment of the present invention, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present invention will not be introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present invention falls within the scope of protection of the present invention.

[0139] Please refer to FIG. 4 , which is a schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention.

[0140] As shown in FIG4 , this embodiment provides a computer-readable storage medium 400 on which a computer program 411 is stored. When the computer program 411 is executed by a processor, the computer program 411 implements a multimodal behavior pattern-based video surveillance early warning method provided by an embodiment of the present invention.

[0141] In a specific implementation process, the computer program 411 can be implemented when executed by a processor. Figure 1 Any implementation manner in the corresponding embodiments.

[0142] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0143] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] The present invention proposes a video surveillance and early warning method based on multimodal behavior patterns, comprising the following steps: extracting multimodal data according to a video stream, performing preprocessing, and aligning and fusing the preprocessed multimodal data, wherein the multimodal data includes RGB frames, optical flow fields, key points of human skeletons, and scene semantic segmentation maps; performing short-term behavior pattern characterization on the fused multimodal data, establishing a behavior pattern library and a scene behavior baseline; updating the dynamic behavior pattern library based on online learning, reducing the weight of historical data in combination with a time attenuation factor, and adjusting the abnormality judgment threshold according to the complexity of the current scene; performing multi-level early warning according to the abnormality judgment threshold, and feedback optimization; the method of the present invention provides an intelligent video surveillance and early warning solution that can automatically learn scene behavior patterns, dynamically adjust the warning threshold, and integrate multi-dimensional feature analysis, thereby solving the problems of high false alarm rate and poor adaptability of the existing technology.

[0145] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A video surveillance and early warning method based on multimodal behavior patterns, characterized in that: It includes the following steps: Extract multimodal data from the video stream, perform preprocessing, and align and fuse the preprocessed multimodal data. The multimodal data includes RGB frames, optical flow fields, human skeleton key points, and scene semantic segmentation maps; Extract short-term behavior pattern features from the fused multimodal data, establish a behavior pattern library, and construct a scene behavior baseline; Update the dynamic behavior pattern library based on online learning, combine the time decay factor to reduce the weight of historical data, and adjust the anomaly determination threshold according to the current scene complexity; Perform multi-level early warning according to the anomaly determination threshold and provide feedback for optimization; The alignment and fusion of the preprocessed multimodal data specifically include: Strictly align the multimodal data through hardware timestamps; Use linear interpolation to complete the intermediate frames for low-frame-rate modalities; Overlay the optical flow field onto the RGB frame channels to obtain the first fused data as five-channel input, and obtain the second fused data by weighting the human skeleton key point trajectories and the semantic segmentation map results through an attention mechanism; The extraction of short-term behavior pattern features from the fused multimodal data, establishment of a behavior pattern library, and construction of a scene behavior baseline specifically are: Use a 3DCNN architecture to extract the spatio-temporal features of the first fused data and the second fused data, which are the short-term behavior pattern features, and construct a behavior pattern library; Stitch the short-term behavior pattern features of the behavior pattern library according to a time window, with a total span of M vectors input into a bidirectional LSTM to output the scene behavior baseline vector; And adjust the anomaly determination threshold according to the current scene complexity, specifically: Extract the parameters of the current scene, including crowd density, light intensity, and motion activity, and perform normalization to obtain the normalized crowd density , light intensity ; Dynamic anomaly determination threshold : ； in, is the basic threshold, is the adjustment coefficient, and the higher the crowd density and the stronger the light intensity, the larger the dynamic anomaly judgment threshold.

2. The multimodal behavior pattern video surveillance early warning method according to claim 1 is characterized in that: Dynamically update the behavior pattern library based on online learning, combine the time decay factor to reduce the weight of historical data, specifically: Obtain the short-term behavior pattern features in real time and perform a sliding window-based incremental update of the behavior pattern library; And set the weights for the historical data in the behavior pattern library according to the generation time. The weight setting specifically is: ； in, is the initial weight, which is 1, is the decay rate, ranging from 0.001 to 0.01; The interval between the current time and the data generation time.

3. The multimodal behavior pattern video surveillance early warning method according to claim 2 is characterized in that: Perform multi-level early warning according to the anomaly determination threshold and provide feedback for optimization, specifically: Obtain the similarity score S(t) between the current behavior feature and the baseline; If S(t) < T(t): Determine it as an anomaly and enter the early warning; If S(t) ≥ T(t) but the administrator marks it as an anomaly: Provide feedback for optimization and trigger incremental learning of the model.

4. A video surveillance and early warning system based on multimodal behavior patterns, characterized in that: It includes: Data processing unit: Extract multimodal data from the video stream, perform preprocessing, and align and fuse the preprocessed multimodal data. The multimodal data includes RGB frames, optical flow fields, human skeleton key points, and scene semantic segmentation maps; Behavior baseline construction unit: Extract short-term behavior pattern features from the fused multimodal data, establish a behavior pattern library, and construct a scene behavior baseline; Update and adjustment unit: Update the dynamic behavior pattern library based on online learning, combine the time decay factor to reduce the weight of historical data, and adjust the anomaly determination threshold according to the current scene complexity; Early warning unit: Perform multi-level early warning according to the anomaly determination threshold and provide feedback for optimization; [[ID= ​ ​ The optical flow field is superimposed on the RGB frame channel to form a five-channel input to obtain the first fusion data, and the human skeleton key point trajectory and semantic segmentation map result are weighted by the attention mechanism to obtain the second fusion data; The fused multimodal data is characterized by short-term behavior patterns, and a behavior pattern library and scenario behavior baseline are established. Specifically: The 3DCNN architecture is used to extract the spatiotemporal features of the first fused data and the second fused data, i.e., the short-term behavior pattern features, and a behavior pattern library is constructed; The short-term behavior pattern features of the behavior pattern library are spliced ​​according to the time window, with a total span of M vectors, which are input into the bidirectional LSTM and the scene behavior baseline vector is output; And adjust the abnormality judgment threshold according to the complexity of the current scene, specifically: Extract the parameters of the current scene, including crowd density, light intensity, and motion activity, and perform normalization to obtain the normalized crowd density , light intensity ; Dynamic anomaly determination threshold : ； in, is the basic threshold, is the adjustment coefficient, and the higher the crowd density and the stronger the light intensity, the larger the dynamic anomaly judgment threshold.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method steps of any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 3 are implemented.

Images