Security behavior event identification method and system based on multi-modal analysis

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, in particular to a security behavior event recognition method and system based on multi-modal analysis, and the method comprises the steps: collecting a multi-modal monitoring data set of a target monitoring scene, the multi-modal monitoring data set comprising a visual imaging data stream and an environment sensing data stream; performing spatial feature extraction processing on the visual imaging data stream to obtain a motion state feature set of the target object; time sequence feature extraction processing is carried out on the environment sensing data stream to obtain an environment state feature set, and the environment state feature set comprises a sound field distribution feature, a temperature gradient feature and an illumination intensity feature; and carrying out space-time fusion processing on the motion state feature set and the environment state feature set to generate a fusion monitoring feature vector, calling an AI security behavior recognition model to carry out event classification processing on the fusion monitoring feature vector, and outputting a security behavior event recognition result of the target monitoring scene.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for identifying security behavior events based on multimodal analysis. Background Art

[0002] With the ever-increasing demand for security, security behavior event analysis technology has become crucial for ensuring safety. In today's society, effective security monitoring is required in all locations. Traditional security behavior analysis primarily relies on a single data source, such as simple video surveillance, which provides limited information about the scene. Specifically, relying solely on a single data source makes it difficult to fully perceive the monitoring scene. For example, visual imaging alone cannot capture important information such as the sound field, temperature, and lighting in the environment, resulting in a one-sided assessment of security behavior events. Furthermore, existing technologies fail to fully integrate the characteristics of different data types and cannot comprehensively analyze data from multiple dimensions of time and space. This results in low accuracy in identifying security behavior events in complex scenarios, making misjudgments and omissions more likely. Summary of the Invention

[0003] In order to at least overcome the above-mentioned deficiencies in the prior art, one of the objectives of the present invention is to provide a method and system for identifying security behavior events based on multimodal analysis.

[0004] An embodiment of the present invention provides a security behavior event recognition method based on multimodal analysis, including: collecting a multimodal monitoring data set of a target monitoring scene, the multimodal monitoring data set including a visual imaging data stream and an environmental sensor data stream; performing spatial feature extraction processing on the visual imaging data stream to obtain a motion state feature set of the target object; performing temporal feature extraction processing on the environmental sensor data stream to obtain an environmental state feature set, the environmental state feature set including sound field distribution features, temperature gradient features and light intensity features; performing spatiotemporal fusion processing on the motion state feature set and the environmental state feature set to generate a fused monitoring feature vector, calling an AI security behavior recognition model to perform event classification processing on the fused monitoring feature vector, and outputting a security behavior event recognition result for the target monitoring scene.

[0005] An embodiment of the present invention also provides a security behavior event recognition system, comprising a processor, a memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the above-mentioned security behavior event recognition method based on multimodal analysis.

[0006] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-mentioned security behavior event identification method based on multimodal analysis.

[0007] The embodiments of the present invention provide a security behavior event recognition method and system based on multimodal analysis. First, a multimodal monitoring data set of the target monitoring scene is collected to provide a rich source of information for a comprehensive understanding of the monitoring scene. Secondly, spatial feature extraction is performed on the visual imaging data stream to accurately obtain the motion state feature set of the target object, which can deeply explore the dynamic situation of the target object in space. Then, time series feature extraction is performed on the environmental sensor data stream to obtain an environmental state feature set including sound field distribution, temperature gradient and light intensity, and then characterize the environmental state change from the time dimension. The motion state feature set and the environmental state feature set are then spatially and temporally fused to generate a fused monitoring feature vector, so that different types of features complement each other to form a comprehensive and unique feature representation. Finally, an AI security behavior recognition model is called to perform event classification processing on the fused monitoring feature vector, which can fully explore the potential relationship between the data and accurately output the security behavior event recognition results of the target monitoring scene, greatly improving the accuracy and comprehensiveness of security behavior event recognition and effectively ensuring the safety of the monitoring scene.

[0008] In summary, the embodiments of the present invention innovatively collect multimodal monitoring data, perform targeted feature extraction and spatiotemporal fusion of different data streams, and then use the AI ​​security behavior recognition model to classify events, which can effectively improve the accuracy of security behavior event recognition and overcome the shortcomings of existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 The present invention provides a flowchart of a method for identifying security behavior events based on multimodal analysis.

[0011] Figure 2 A block diagram of a security behavior event recognition system provided by an embodiment of the present invention.

[0012] icon: 100-Security behavior event recognition system; 101 - processor; 102 - memory; 103 - bus. DETAILED DESCRIPTION

[0013] The exemplary embodiments disclosed herein will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0014] In order to better understand the above technical solution, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0015] Figure 1 This is a flowchart of a security behavior event recognition method based on multimodal analysis according to an embodiment of the present invention, which is applied to a security behavior event recognition system and includes steps 110 to 140.

[0016] Step 110: Collect a multimodal monitoring data set of the target monitoring scene, where the multimodal monitoring data set includes a visual imaging data stream and an environmental sensing data stream.

[0017] In an embodiment of the present invention, a public area of ​​a large commercial plaza is used as the target monitoring scene. Multiple high-definition cameras are deployed in the area to collect visual imaging data streams. These cameras are distributed in different locations and can cover every corner of the plaza. At the same time, a variety of environmental sensors are arranged, including microphones for collecting sound field intensity data, temperature sensors for obtaining temperature gradient data, and light sensors for collecting light intensity data, thereby forming an environmental sensing data stream. For example, the camera continuously captures video images at a fixed frame rate to form a continuous visual imaging data stream; the microphone captures the surrounding sound in real time, the temperature sensor records temperature data at regular intervals, and the light sensor monitors changes in light intensity at any time. These data sources are continuously collected and together constitute a multimodal monitoring data set. Then, these data are transmitted to the back-end data processing center for subsequent processing and analysis.

[0018] Step 120: Perform spatial feature extraction processing on the visual imaging data stream to obtain a motion state feature set of the target object.

[0019] In this large commercial plaza scenario, spatial feature extraction is performed on the collected visual imaging data stream. By analyzing the visual imaging data stream, a set of motion state features of target objects (such as people moving in the plaza) is obtained.

[0020] In an optional embodiment, performing spatial feature extraction processing on the visual imaging data stream to obtain a motion state feature set of the target object includes: Step 121: parse the visual imaging data stream frame by frame to extract the pixel distribution area of ​​the target object in the continuous frame sequence.

[0021] In the visual imaging data stream of a commercial plaza, the video is parsed frame by frame at a rate of several frames per second. For example, 30 frames of video are collected per second, and each frame is analyzed in detail. Image recognition technology is used to identify target objects, such as people, in the image. For each target object, its pixel position in the image is determined, and then its pixel distribution area is determined. Taking a person walking in the plaza as an example, in a certain frame, the image analysis algorithm determines the pixel range occupied by the person from head to toe, forming a pixel distribution area containing the person's outline. As the video frames advance, multiple frames are continuously analyzed to obtain the pixel distribution area of ​​the target object in a series of consecutive frame sequences, providing basic data for subsequent analysis.

[0022] Step 122: Perform contour detection on the pixel distribution area to generate a posture contour boundary of the target object.

[0023] A contour detection algorithm is used to process the pixel distribution areas of the target object in the continuous frame sequence obtained in the previous step. For example, a classic edge detection algorithm is used to analyze each pixel distribution area and identify boundaries where pixel values ​​change significantly. For a person walking in the square, the contour detection algorithm can clearly outline the person's body and determine their posture contour boundaries. This posture contour boundary can reflect the person's external contour at different times, such as standing, walking, and bending.

[0024] Step 123: Generate a three-dimensional motion trajectory model based on the deformation parameters of the posture contour boundary, and extract the displacement rate and direction change of each key node in the three-dimensional motion trajectory model.

[0025] Optionally, based on the obtained posture contour boundary, its deformation between different frames is analyzed. For example, by comparing the posture contour boundaries of adjacent frames, the position changes of key points on the contour boundary are calculated, and these change parameters are used to generate a three-dimensional motion trajectory model. In the three-dimensional space of the commercial square, the motion trajectory of the target object in the three-dimensional space is determined by combining the position information of the camera and the proportional relationship in the picture. For people walking in the square, their motion trajectory model in the three-dimensional space is generated by analyzing the deformation of the posture contour boundary. In this model, some key nodes are selected, such as the points where people change their walking direction or speed at different positions, and the displacement rate and direction change at these key nodes are calculated. The displacement rate can be obtained by calculating the ratio of the distance change between the key nodes to the time interval, and the direction change is determined by analyzing the angle of change in the motion direction at the key nodes.

[0026] Step 124: performing window sliding average processing on the displacement rate and the direction change to generate a smoothed motion trajectory feature.

[0027] To reduce data noise and fluctuations, a window sliding average is applied to the extracted displacement rate and directional change. For example, a fixed-length time window is set to span 10 frames. Within this window, the displacement rate and directional change are averaged. For a person walking in the square, within a specific 10-frame window, the average displacement rate and directional change are calculated. As the window slides frame by frame along the motion trajectory, the average calculation is continuously performed, resulting in a series of smoothed displacement rate and directional change data. These data constitute the smoothed motion trajectory feature, which can more stably reflect the motion trend of the target object.

[0028] Step 125: Calling a pre-trained posture analysis model to identify the joint angle changes of the posture contour boundary, and extracting the limb extension amplitude characteristics and movement frequency characteristics of the target object.

[0029] Optionally, a pre-trained posture analysis model is used. The model has been trained on a large amount of posture data and can accurately identify the changes in joint angles at the boundaries of posture contours. For people moving in a commercial plaza, the posture analysis model is used to analyze the angle changes of points representing joint positions on the boundaries of the posture contours. For example, the angle changes of the joints of the arms and legs of the person are analyzed to extract the characteristics of the limb extension amplitude. The limb extension amplitude can be determined by calculating the range of change of the joint angle. At the same time, the number of joint angle changes per unit time is counted to obtain the motion frequency characteristics. For example, within one minute, the number of times a person swings his arms is counted as a measure of the motion frequency.

[0030] Step 126: Obtain the motion state feature set according to the smoothed motion trajectory feature, the limb extension amplitude feature, and the action frequency feature.

[0031] Furthermore, the processed smoothed motion trajectory features, limb extension amplitude features, and movement frequency features are integrated. These features describe the target object's motion state in the visual imaging data stream from different aspects. In the commercial plaza scene, for each monitored target object, its smoothed motion trajectory features (such as average displacement rate and stable movement direction change trend), limb extension amplitude features (such as the extension range of arms and legs), and movement frequency features (such as the number of movements per minute) are combined to form a complete motion state feature set. This set comprehensively reflects the target object's motion state in visual space and provides a foundation for subsequent fusion with the environmental state feature set.

[0032] Step 130: performing time series feature extraction processing on the environmental sensor data stream to obtain an environmental state feature set, wherein the environmental state feature set includes sound field distribution features, temperature gradient features, and light intensity features.

[0033] In the scenario of a large commercial plaza, the collected environmental sensor data stream is processed by time series feature extraction to obtain the environmental state feature set.

[0034] In another preferred embodiment, the performing of time series feature extraction processing on the environmental sensor data stream to obtain an environmental state feature set includes: Step 131: performing sampling rate alignment processing on the environmental sensing data stream, and converting the sound field intensity data, temperature gradient data and light intensity data of different frequencies into a synchronous data sequence with a unified time stamp.

[0035] In a commercial plaza, different types of environmental sensors may collect data at different frequencies. For example, sound field intensity data may be collected 100 times per second, temperature gradient data 10 times per second, and light intensity data 5 times per second. To facilitate subsequent unified processing, these data with different frequencies need to be sampled at the same rate. Using methods such as interpolation or downsampling, all data is converted into a synchronized data sequence with a unified timestamp. For example, converting sound field intensity data, temperature gradient data, and light intensity data to a frequency of 10 times per second and marking each data point with the same timestamp allows these different types of data to be aligned in the time dimension, forming a synchronized data sequence for subsequent analysis.

[0036] Step 132: performing spectrum decomposition processing on the sound field intensity data in the synchronous data sequence, and extracting the energy distribution ratio characteristics of each frequency band as the sound field distribution characteristics.

[0037] Optionally, for the sound field intensity data in the synchronized data sequence after sampling rate alignment, a spectrum decomposition algorithm, such as Fourier transform, is used to convert the time domain sound field intensity signal into the frequency domain. In the frequency domain, the energy distribution of different frequency bands is analyzed. For example, the sound frequency range is divided into multiple frequency bands, such as low frequency band (20Hz-200Hz), mid-frequency band (200Hz-2000Hz), high frequency band (2000Hz-20000Hz), etc. The proportion of energy in each frequency band to the total energy is calculated, and these proportion values ​​constitute the sound field distribution characteristics. In a commercial plaza, different activities or events may result in different sound field distribution characteristics. For example, the energy in the low frequency band may increase when a crowd gathers, and the energy in the mid- and high-frequency bands will be more prominent during music performances.

[0038] Step 133: performing spatial interpolation processing on the temperature gradient data to generate a two-dimensional temperature field distribution map, and extracting the thermal gradient direction characteristics and the temperature difference change rate characteristics in the two-dimensional temperature field distribution map as the temperature gradient characteristics.

[0039] Optionally, for the temperature gradient data in the synchronized data sequence, since the distribution of sensors is discrete, in order to more comprehensively understand the temperature distribution, a spatial interpolation algorithm, such as Kriging interpolation, is used to generate a two-dimensional temperature field distribution map based on the discrete temperature sensor data. In this distribution map, the direction of the thermal gradient, that is, the direction in which the temperature changes fastest, is analyzed. At the same time, the rate of change of the temperature difference between adjacent time points is calculated. For example, by comparing the two-dimensional temperature field distribution maps of two adjacent seconds, the temperature change at each location is calculated, and then the temperature difference change rate is obtained. These thermal gradient direction characteristics and temperature difference change rate characteristics constitute the temperature gradient characteristics, which can reflect the dynamic changes in temperature within the commercial plaza.

[0040] Step 134: performing abnormal fluctuation detection processing on the light intensity data, and identifying the duration and fluctuation amplitude of the sudden change in light intensity as the light intensity feature.

[0041] Optionally, abnormal fluctuation detection is performed on the light intensity data in the synchronized data sequence. By setting appropriate thresholds and algorithms, sudden changes in light intensity can be identified. For example, a sudden change in light intensity occurs when the light intensity suddenly increases or decreases by more than a certain percentage within a short period of time. The duration of the sudden change is recorded, that is, the length of time from the start of the sudden change in light intensity to the return to normal or the reaching of a new stable state. At the same time, the fluctuation amplitude is calculated, that is, the difference between the maximum and minimum light intensity values ​​during the sudden change. These durations and fluctuation amplitudes serve as light intensity characteristics that can reflect abnormal changes in lighting conditions within the commercial plaza.

[0042] Step 135: Generate an environmental state feature set based on the sound field distribution feature, the temperature gradient feature, and the light intensity feature.

[0043] The extracted sound field distribution features (energy distribution ratio in each frequency band), temperature gradient features (thermal gradient direction and temperature difference change rate), and light intensity features (sudden duration and fluctuation amplitude) are further integrated. In the commercial plaza scenario, these different environmental features are combined at each time point to form a complete environmental state feature set. This set comprehensively reflects the environmental state of the commercial plaza at that moment, providing environmental information for subsequent spatiotemporal fusion with the motion state feature set.

[0044] Step 140: Perform spatiotemporal fusion processing on the motion state feature set and the environmental state feature set to generate a fused monitoring feature vector, call the AI ​​security behavior recognition model to perform event classification processing on the fused monitoring feature vector, and output the security behavior event recognition result of the target monitoring scene.

[0045] In the monitoring scenario of a large commercial plaza, the motion state feature set and the environmental state feature set are spatially and temporally fused to obtain more comprehensive information, and the AI ​​security behavior recognition model is used to classify events.

[0046] In another optional embodiment, performing spatiotemporal fusion processing on the motion state feature set and the environment state feature set to generate a fused monitoring feature vector includes: Step 141: performing time stamp processing on the smoothed motion trajectory features in the motion state feature set to generate a first time series feature sequence; performing time stamp processing on the energy distribution ratio features in the environment state feature set to generate a second time series feature sequence.

[0047] It can be understood that for the smoothed motion trajectory features in the motion state feature set, a timestamp is added to each data point. For example, according to the time sequence of the collected data, the corresponding time is marked for each smoothed motion trajectory feature data point to form a first time series feature sequence, which reflects the change of the motion state of the target object over time. For the energy distribution ratio feature (which is part of the sound field distribution feature) in the environmental state feature set, a timestamp is also marked for each data point to generate a second time series feature sequence, which shows the change of the sound field distribution feature in the time dimension. Through this timestamp marking process, the motion state features and the environmental state features are comparable in time, preparing for the subsequent spatiotemporal alignment.

[0048] Step 142: Input the first time series feature sequence and the second time series feature sequence into a spatiotemporal alignment model to determine a time offset between the smoothed motion trajectory feature and the energy distribution ratio feature.

[0049] Optionally, the first and second time series feature sequences are input into a spatiotemporal alignment model. This model determines the time offset between the smoothed motion trajectory features and the energy distribution ratio features by analyzing the temporal correlation and feature variation patterns of the data in the two sequences. For example, in a commercial plaza, a time delay may be observed between certain changes in motion state (such as a sudden gathering of people) and changes in sound field distribution characteristics (such as a sudden increase in sound intensity). The spatiotemporal alignment model calculates this time offset through a detailed analysis of the two time series feature sequences, allowing subsequent compensation of the motion state features to better align them with the environmental state features in time.

[0050] Step 143: performing interpolation compensation processing on the first time series feature sequence based on the time offset to generate a compensated motion trajectory feature sequence.

[0051] Optionally, interpolation compensation is performed on the first time series feature sequence based on the time offset determined in the previous step. For example, if the time offset indicates that the motion state feature lags behind the environmental state feature in time, then in the first time series feature sequence, some intermediate data points are inserted at appropriate locations based on the size of the time offset. The values ​​of these inserted points are calculated using an interpolation algorithm (such as linear interpolation) to make the motion state feature sequence more synchronized in time with the environmental state feature sequence. The resulting compensated motion trajectory feature sequence can be better integrated with the environmental state feature.

[0052] Step 144: Input the compensated motion trajectory feature sequence, the limb extension amplitude feature, the movement frequency feature, the thermal gradient direction feature, the temperature difference change rate feature, the duration and the fluctuation amplitude into the feature splicing layer for dimensional alignment processing to generate a fusion monitoring feature vector.

[0053] Optionally, the processed compensated motion trajectory feature sequence, limb extension amplitude features, movement frequency features (from the motion state feature set), thermal gradient direction features, temperature difference change rate features (from the temperature gradient features, part of the environmental state feature set), and the duration and fluctuation amplitude of sudden light intensity changes (from the environmental state feature set) are input into the feature concatenation layer. In the feature concatenation layer, these features of different dimensions are processed to align their dimensions. For example, some feature vectors of different dimensions are adjusted to the same dimension through operations such as zero padding or dimensional expansion. They are then concatenated to form a fused monitoring feature vector. This vector incorporates multiple aspects of motion state and environmental state information, providing a rich data foundation for subsequent event classification.

[0054] In a preferred embodiment, the calling of the AI ​​security behavior recognition model to perform event classification processing on the fused monitoring feature vector and outputting the security behavior event recognition result of the target monitoring scene includes: Step 145: Input the fused monitoring feature vector into the feature coding layer of the AI ​​security behavior recognition model to generate a multi-dimensional abstract feature representation.

[0055] Optionally, the generated fused monitoring feature vector is input into the feature encoding layer of the AI ​​security behavior recognition model. The feature encoding layer extracts and abstracts the features of the fused monitoring feature vector through a series of neural network operations, such as convolution and pooling. For example, a sliding convolution operation is performed on the fused monitoring feature vector using a convolution kernel to extract feature information from different local regions. The features are then compressed and aggregated through a pooling operation to generate a multi-dimensional abstract feature representation. This multi-dimensional abstract feature representation can more efficiently represent the information in the fused monitoring feature vector, providing more concise and representative data for subsequent analysis.

[0056] Step 146: Perform multi-scale convolution processing on the multi-dimensional abstract feature representation to extract local correlation features and global distribution features under different receptive fields.

[0057] Optionally, multi-scale convolution processing is performed on the generated multi-dimensional abstract feature representation. Convolution kernels of different sizes are used to perform convolution operations on the multi-dimensional abstract feature representation. Smaller convolution kernels can capture detailed information of local areas and extract local correlation features; larger convolution kernels can cover wider areas and extract global distribution features. For example, using a 3×3 convolution kernel can extract local feature associations, such as the relationship between a person's action and the surrounding small-scale environmental features; using a 7×7 convolution kernel can extract more global feature distributions, such as the overall relationship between the movement of people and the environmental status in the entire commercial plaza. Through multi-scale convolution processing, multi-dimensional abstract feature representations can be analyzed from different angles and scales to obtain more comprehensive information.

[0058] Step 147: Input the local correlation features and the global distribution features into the attention weight allocation layer to generate feature importance weights.

[0059] Optionally, the extracted local correlation features and global distribution features are input into the attention weight allocation layer, which calculates the importance of each feature and assigns corresponding weights to different features. For example, by analyzing the correlation between local correlation features and global distribution features and security behavior events, a predetermined calculation method (such as neural network-based weight calculation) is used to determine which features are more important for identifying corresponding security behavior events. These important features are assigned higher weights, while relatively unimportant features are assigned lower weights. These weights reflect the importance of each feature in event classification.

[0060] Step 148: Perform weighted fusion processing on the local correlation features and the global distribution features based on the feature importance weights to generate an optimized feature representation.

[0061] Based on the generated feature importance weights, a weighted fusion process is performed on the local correlation features and the global distribution features. Each local correlation feature and the global distribution feature is multiplied by the corresponding weight, and then added together (or a suitable fusion method is used, such as concatenating and then performing a weighted summation) to generate an optimized feature representation. This way, features with higher importance are more reflected in the fusion results, allowing the optimized feature representation to more accurately reflect information related to security behavior events and improve the accuracy of event classification.

[0062] Step 149: Input the optimized feature representation to the fully connected classification layer, and output the security behavior event recognition result, which includes normal behavior categories, potential risk behavior categories, and emergency threat behavior categories.

[0063] Optionally, the optimized feature representation is input into a fully connected classification layer. The fully connected classification layer calculates and classifies the optimized feature representation through a series of weight matrices and activation functions. According to the predefined classification criteria, the input feature representation is mapped to different security behavior event categories, and the normal behavior category, potential risk behavior category, and emergency threat behavior category are output. For example, if the optimized feature representation shows that the movement state of the personnel and the environmental state are both within the normal range and there are no abnormal features, the classification result may be the normal behavior category; if some abnormal movement patterns (such as fast running, gathering) are found in combination with different environmental anomalies (such as sudden strong light flashes, abnormal sounds), it may be classified as the potential risk behavior category; if serious abnormalities occur (such as violent behavior, fire-related environmental changes), it will be classified as the emergency threat behavior category.

[0064] In an alternative embodiment, the training process of the AI ​​security behavior recognition model includes: Step 210: Acquire a multimodal training data set of historical monitoring scenes, wherein the multimodal training data set includes a visual training data stream and an environmental training data stream annotated with security behavior event categories.

[0065] In this embodiment of the present invention, a large commercial plaza is still used as an example. Multimodal training data is collected from surveillance records of the plaza over a period of time. The visual training data stream is composed of videos recorded by various cameras at different time periods. These videos cover various activity scenes in the plaza, such as normal daily traffic, crowds during store promotions, and occasional conflicts. The environmental training data stream is collected by environmental sensors distributed throughout the plaza during corresponding time periods. It includes information such as sound field intensity, temperature changes, and light intensity fluctuations during different activities. At the same time, a professional annotation team accurately annotates each set of visual training data streams and the corresponding environmental training data stream with security behavior event categories based on the video content and actual occurrences. For example, normal human movement is labeled as normal behavior, abnormal crowds are labeled as potential risk behavior, and sudden violent conflicts are labeled as urgent threat behavior. This labeled data provides an accurate reference for subsequent model training, enabling the model to learn the correct classification corresponding to different behaviors and environmental states. Next, these labeled visual training data streams and environmental training data streams are integrated together to form a complete multimodal training data set, which is stored in a dedicated database for subsequent use in training AI security behavior recognition models.

[0066] Step 220: Perform motion state feature extraction processing on the visual training data stream to obtain a training motion state feature set.

[0067] Using historical surveillance data from a large commercial plaza, motion state feature extraction is performed on the visual training data stream. A similar method to that used in step 120 is used to obtain a training motion state feature set from the visual training data stream. First, the visual training data stream is parsed frame by frame. For example, a video recording one hour of activity in a specific area of ​​the plaza is analyzed frame by frame at a fixed frame rate (e.g., 30 frames per second). Advanced image recognition algorithms are used to accurately identify the target object (person) in each frame and determine its pixel distribution area. As the frames progress, a series of pixel distribution areas for the target object in the continuous frame sequence is obtained.

[0068] Next, these pixel distribution areas are processed for contour detection. Using an edge detection algorithm, the posture contour boundaries of the target object in different frames are accurately outlined, clearly presenting the body posture contours of the person at each moment. Based on the deformation parameters of these posture contour boundaries, a three-dimensional motion trajectory model is generated. Combining the three-dimensional spatial information of the square and the parameters of the camera, the motion trajectory of the target object in three-dimensional space is determined. From this three-dimensional motion trajectory model, the displacement rate and direction change of each key node are extracted. For example, when a person suddenly changes the walking direction or the speed changes significantly at a certain location, these locations are key nodes, and the displacement rate and direction change at these key nodes are calculated.

[0069] Afterwards, a window sliding average is applied to the displacement rate and directional change. An appropriate window length is set (e.g., the time length corresponding to 10 frames), and the average value is calculated within this window. As the window slides along the trajectory, smoothed motion trajectory features are obtained. Simultaneously, a pre-trained posture analysis model is used to identify joint angle changes at the posture contour boundary, extracting the target object's limb extension amplitude characteristics and movement frequency characteristics. For example, the range of arm joint angle changes is analyzed to obtain limb extension amplitude characteristics, and the number of joint angle changes per unit time (e.g., one minute) is counted to obtain movement frequency characteristics.

[0070] Finally, the smoothed motion trajectory features, limb extension amplitude features, and movement frequency features are integrated together to form a training motion state feature set. This set comprehensively and meticulously describes the motion state of the target object in the visual training data stream, providing key motion information for subsequent fusion with the environmental state feature set and model training.

[0071] Step 230: Perform time series feature extraction processing on the environmental training data stream to obtain a training environmental state feature set.

[0072] In the historical data processing of the commercial plaza, time series feature extraction is performed on the environmental training data stream to obtain a set of environmental state features for training. First, the sampling rate of the environmental training data stream is aligned. Since different types of environmental sensors collect data at different frequencies, for example, the sound field intensity sensor may collect data 50 times per second, the temperature sensor collects data 10 times per second, and the light intensity sensor collects data 5 times per second. In order to unify data processing, appropriate interpolation or downsampling methods are used to convert all data into a synchronized data sequence with a unified timestamp. All data are adjusted to a frequency of 10 times per second, and the corresponding timestamp is accurately marked for each data point.

[0073] Next, the sound field intensity data in the synchronized data sequence undergoes spectral decomposition. Using spectral analysis algorithms such as Fourier transform, the time-domain sound field intensity signal is converted to the frequency domain, analyzing the energy distribution in different frequency bands. The sound frequency range is divided into multiple frequency bands, such as 0-100 Hz for low frequency, 100-1000 Hz for mid-frequency, and 1000-10000 Hz for high frequency. The proportion of energy in each frequency band to the total energy is calculated, and these proportions constitute the sound field distribution characteristics. Spatial interpolation is performed on the temperature gradient data. Because the temperature sensors are discretely distributed within the square, spatial interpolation algorithms such as Kriging interpolation are used to generate a two-dimensional temperature field distribution map based on this discrete temperature data. From this distribution map, the thermal gradient direction (i.e., the direction of the fastest temperature change) and the temperature difference change rate are extracted. These characteristics are calculated by comparing the temperature field distribution maps at adjacent time points. Abnormal fluctuation detection is performed on the light intensity data. An appropriate threshold is set. When the light intensity change exceeds this threshold within a short period of time, an abnormal fluctuation is detected. Record the duration of the abnormal fluctuation, that is, the length of time from the beginning to the end of the fluctuation, and the amplitude of the fluctuation, that is, the difference between the maximum and minimum light intensity during the fluctuation.

[0074] Finally, based on the sound field distribution characteristics, temperature gradient characteristics and light intensity characteristics, a training environment state feature set is generated. This set comprehensively reflects the temporal changes of the environment state in the historical monitoring data, and provides important environmental information for the subsequent fusion with the motion state feature set and model training.

[0075] Step 240: Perform spatiotemporal fusion processing on the training motion state feature set and the training environment state feature set to generate a training fused monitoring feature vector.

[0076] In the training data processing phase of the large commercial plaza, the training motion state feature set and the training environment state feature set are spatiotemporally fused to generate a training fused monitoring feature vector. First, the smoothed motion trajectory features in the training motion state feature set are timestamped. According to the time sequence of data acquisition, the accurate time is marked for each smoothed motion trajectory feature data point to form a first time series feature sequence. This sequence clearly shows the changes in the motion state of the target object over time. At the same time, the energy distribution ratio features in the sound field distribution features in the training environment state feature set are timestamped to generate a second time series feature sequence. This sequence reflects the changes in the sound field distribution features in the time dimension.

[0077] Then, the first time series feature sequence and the second time series feature sequence are input into the spatiotemporal alignment model. The spatiotemporal alignment model determines the time offset between the smoothed motion trajectory feature and the energy distribution ratio feature by performing a detailed analysis of the two sequences and comparing the temporal correlation of the data in the sequences and the law of feature changes. For example, it is found that in some events, the change in the motion state of the person lags slightly behind the change in the sound field distribution feature in time, and the time offset is calculated by the model. Based on the time offset, the first time series feature sequence is interpolated and compensated. If the time offset indicates that the motion state feature lags, an intermediate data point is inserted at the appropriate position of the first time series feature sequence, and the value of the insertion point is calculated by an algorithm such as linear interpolation, so that the motion state feature sequence is more temporally matched with the environmental state feature sequence, and a compensated motion trajectory feature sequence is generated.

[0078] Finally, the compensated motion trajectory feature sequence, the limb extension amplitude features and movement frequency features in the training motion state feature set, and the thermal gradient direction features, temperature difference change rate features, and the duration and amplitude of abnormal light intensity fluctuations in the training environmental state feature set are input together into the feature splicing layer. In the feature splicing layer, these features of different dimensions are dimensionally aligned. For feature vectors of different dimensions, they are adjusted to the same dimension by padding with zeros or performing dimensional expansion operations, and then spliced ​​in a certain order to form a fused monitoring feature vector for training. This vector integrates multiple aspects of information about motion state and environmental state in time and space, providing a rich and accurate data foundation for the training of AI security behavior recognition models.

[0079] Step 250: Obtain an initial neural network model, input the training fusion monitoring feature vector into the initial neural network model for forward propagation processing, and obtain a predicted security behavior event category.

[0080] During the training process of the AI ​​security behavior recognition model, an initial neural network model is first obtained. The initial model has a certain network structure, such as multiple hidden layers and output layers. The hidden layer is used to extract features and perform nonlinear transformations on the input data, and the output layer is used to generate prediction results.

[0081] The generated fused monitoring feature vector for training is then input into the initial neural network model. Within the model, data propagates forward, starting from the input layer. After receiving the fused monitoring feature vector, the input layer passes it to the first hidden layer. Neurons in the hidden layer process the input data through weighted summation and activation functions. For example, a neuron multiplies the input feature vector by the corresponding weight matrix, adds a bias term, and then performs a nonlinear transformation using an activation function (such as the ReLU function). This yields a feature representation after processing in the first layer. This feature representation is then passed to the next hidden layer, and the above process is repeated to continuously extract and transform features.

[0082] After processing through multiple hidden layers, the data ultimately reaches the output layer. The output layer multiplies the features extracted and processed by the previous hidden layers with the output layer's weight matrix and adds a bias term to produce the predicted security behavior event category. The output of the output layer is a probability distribution vector, with each element corresponding to the probability of a security behavior event category (e.g., normal behavior, potential risk behavior, and emergency threat behavior). The category with the highest probability is the predicted security behavior event category. Through this forward propagation process, the initial neural network model analyzes and predicts the fused monitoring feature vector used for training, providing a foundation for subsequent model training and optimization.

[0083] Step 260: Calculate the cross entropy loss between the predicted security behavior event category and the labeled security behavior event category, and update the weight parameters of the initial neural network model based on the cross entropy loss until the loss converges to obtain an AI security behavior recognition model.

[0084] After obtaining the predicted security behavior event category, the cross-entropy loss between the predicted result and the annotated security behavior event category needs to be calculated. First, a category probability distribution vector is generated based on the predicted security behavior event category. Each element in this vector represents the predicted probability value for each security behavior event category. For example, if the prediction result shows a probability of 0.6 for the normal behavior category, 0.3 for the potential risk behavior category, and 0.1 for the emergency threat category, the category probability distribution vector is [0.6, 0.3, 0.1]. Next, the annotated security behavior event category is one-hot encoded to generate a target category probability distribution vector. For example, if the annotated category is the potential risk behavior category, the one-hot encoded target category probability distribution vector is [0, 1, 0]. Next, the KL divergence (KL divergence is a metric that measures the difference between two probability distributions) between the category probability distribution vector and the target category probability distribution vector is calculated to generate an initial loss value. This initial loss value is then weighted to increase the loss contribution of scarce categories. In security behavior event data, certain categories (such as emergency threat behavior categories) may appear less frequently. In order to make the model pay more attention to these scarce categories, their corresponding loss values ​​are given higher weights, thereby obtaining adjusted loss values.

[0085] Finally, the adjusted loss value is smoothed and regularized to generate the cross-entropy loss. This cross-entropy loss measures the degree of discrepancy between the model's predictions and the true annotations. Based on this cross-entropy loss, its gradient with respect to the weight parameters of each layer of the initial neural network model is calculated. The gradient value represents the direction and rate of change of the loss function under the current weight parameters. The gradient value is then subjected to an adaptive learning rate adjustment process, dynamically adjusting the learning rate based on the current training situation to generate an optimized gradient update direction. The weight parameters are iteratively updated according to this optimized gradient update direction, with each update adjusting the weight parameters in a direction that reduces the cross-entropy loss.

[0086] After each iterative update, the weight parameters are clipped to limit their range of variation and prevent model instability caused by excessively large or small weights. This process continues until the loss value decreases by less than a preset threshold over multiple iterations, indicating that the loss has converged. At this point, the difference between the model's predictions and annotations is sufficiently small, and the training process terminates. The current weight parameters are saved, resulting in an AI security behavior recognition model. This trained and optimized model can more accurately classify events based on the fused monitoring feature vectors and output reliable security behavior event recognition results.

[0087] In an optional embodiment, the calculating the cross entropy loss between the predicted security behavior event category and the labeled security behavior event category includes: Step 261: Generate a category probability distribution vector based on the predicted security behavior event category; perform one-hot encoding on the labeled security behavior event category to generate a target category probability distribution vector; calculate the KL divergence between the category probability distribution vector and the target category probability distribution vector to generate an initial loss value; perform category weight adjustment on the initial loss value to increase the loss contribution ratio of the scarce category to obtain an adjusted loss value; perform smoothing regularization on the adjusted loss value to generate the cross entropy loss.

[0088] Taking a specific training scenario in a large commercial plaza as an example, after the model performs forward propagation on a set of fused monitoring feature vectors used for training, it obtains the predicted security behavior event category. For example, if the prediction results show a probability of 0.5 for the normal behavior category, 0.3 for the potential risk behavior category, and 0.2 for the emergency threat category, then a category probability distribution vector P = [0.5, 0.3, 0.2] is generated based on this prediction result. At the same time, the labeled security behavior event category corresponding to this set of training data is checked. For example, if it is labeled as the emergency threat category, this labeled category is one-hot encoded to generate the target category probability distribution vector Q = [0, 0, 1].

[0089] Next, the KL divergence between the class probability distribution vector P and the target class probability distribution vector Q is calculated. Using the KL divergence formula, the initial loss value is calculated by summing the elements of the two vectors. For example, the initial loss value is calculated to be a specific value (set to 0.8). Since the emergency threat behavior category is relatively rare in the commercial plaza security data, it is considered a scarce category. To make the model focus more on this scarce category, the initial loss value is adjusted by category weighting. A weight coefficient is set: 2 for the emergency threat behavior category and 1 for all other categories. Therefore, for this example, the adjusted loss value = 0 (adjusted loss for the normal behavior category, due to a weight coefficient of 1 and a probability of 0) + 0 (adjusted loss for the potential risk behavior category, due to a weight coefficient of 1 and a probability of 0) + 0.8 × 2 (adjusted loss for the emergency threat behavior category, due to a weight coefficient of 2 and an initial loss of 0.8), resulting in an adjusted loss value of 1.6.

[0090] Finally, the adjusted loss value is smoothed and regularized. Through some smoothing regularization algorithms, such as adding regularization terms, the adjusted loss value is further processed to generate the cross-entropy loss. This cross-entropy loss can more accurately reflect the difference between the model prediction and the true annotation, providing a more effective basis for subsequent updates to the model weight parameters.

[0091] As another optional implementation, updating the weight parameters of the initial neural network model based on the cross entropy loss until the loss converges to obtain the AI ​​security behavior recognition model includes: Step 262: Calculate the gradient value of the cross entropy loss with respect to the weight parameters of each layer of the initial neural network model; perform adaptive learning rate adjustment processing on the gradient value to generate an optimized gradient update direction; iteratively update the weight parameters according to the optimized gradient update direction; perform amplitude clipping processing on the weight parameters after each iterative update to limit the range of change of the weight parameters; when the loss value decreases by less than a preset threshold for multiple consecutive iterations, it is determined that the loss convergence state is reached, the training process is terminated, and the current weight parameters are saved to obtain the AI ​​security behavior recognition model.

[0092] During the commercial plaza model training process, after calculating the cross-entropy loss, the gradient of this loss with respect to the weight parameters of each layer of the initial neural network model is first calculated. Using the backpropagation algorithm, starting from the output layer, the loss value is propagated backward through the network structure, calculating the gradient of the loss with respect to each weight parameter of each layer. For example, for a certain weight parameter in the hidden layer, the partial derivative of the loss function with respect to this weight parameter is calculated to obtain its gradient value. These gradient values ​​indicate how the loss value would change if the weight parameter was adjusted.

[0093] Next, these gradient values ​​are adaptively adjusted using the learning rate. The adaptive learning rate algorithm dynamically adjusts the learning rate based on the current training situation. For example, if the loss value decreases rapidly over several consecutive iterations, this indicates that the current learning rate may be too high, and the adaptive learning rate algorithm will appropriately decrease the learning rate. Conversely, if the loss value decreases slowly, the learning rate may be increased. This generates an optimized gradient update direction. Based on this optimized gradient update direction, the weight parameters are iteratively updated. For example, for a weight parameter W, the new weight parameter W' = W - learning rate × gradient value, where the learning rate is the adaptively adjusted learning rate.

[0094] After each iterative update, the weight parameters are clipped. A maximum and minimum amplitude limit is set, with the maximum amplitude being 1 and the minimum amplitude being -1. If the updated weight parameters exceed this range, they are clipped to the maximum or minimum amplitude value to limit the range of variation of the weight parameters and prevent model instability caused by excessively large or small weight parameters. The above weight parameter update process is continued, and the cross-entropy loss is calculated after each iteration. When the loss value decreases by less than a preset threshold (set to 0.01) over multiple consecutive iterations (set to 10 times), it is determined that the loss has reached a state of convergence. At this point, the model's predictive performance has basically stabilized, the training process is terminated, and the current weight parameters are saved. These saved weight parameters determine the final network structure and parameter configuration of the AI ​​security behavior recognition model, enabling the model to accurately classify events based on the input fused monitoring feature vector and output reliable security behavior event recognition results.

[0095] In one example, the multimodal training data set further includes a spatial training data stream, and the method further includes: Step 2100: Perform random frame sampling processing on the visual training data stream to generate visual training subsets with different temporal resolutions; perform noise injection processing on the environmental training data stream to simulate training data under different environmental interference conditions; perform coordinate perturbation processing on the spatial training data stream to generate training samples with spatial position offset; merge the visual training subset, the environmental training data stream after the noise injection processing, and the spatial training data stream after the coordinate perturbation processing to generate an enhanced multimodal training data set.

[0096] In the training data augmentation scenario of a large commercial plaza, random frame sampling is performed on the visual training data stream. For example, from a long visual training video, samples are taken at varying frame intervals according to a random rule. Some sampling intervals might be 5 frames, others 10 frames, and so on. This method generates visual training subsets with different temporal resolutions. These subsets contain information from different time points in the video, enabling the model to learn the motion characteristics of the target object at different time scales. Noise injection is performed on the environmental training data stream. Random noise signals are added to the sound field intensity data to simulate noise interference that can occur in real environments. For example, small random noise with a certain probability is added to the originally stable sound field intensity data to introduce some fluctuations in the data, similar to the effect of occasional noisy background noise in a commercial plaza on sound field intensity monitoring. For temperature gradient data, small random deviations are added to the measured values ​​to simulate measurement errors or minor temperature fluctuations in the environment. For light intensity data, the values ​​of some data points are randomly altered to simulate possible interference with the light sensor or sudden changes in light and shadow in the environment.

[0097] Coordinate perturbations are performed on the spatial training data stream. The spatial training data stream records the coordinate information and related spatial features of key locations within the square. These coordinates are then randomly and slightly offset. For example, the coordinates of a certain location are increased or decreased by a small random value on the x-axis, and similar small perturbations are performed on the y-axis. This generates training samples with spatial position shifts, allowing the model to learn the potential impact of different spatial position changes on security behavior events.

[0098] Finally, the processed visual training subset, the noise-injected environmental training data stream, and the coordinate-perturbation-processed spatial training data stream are combined. This enhanced data from various sources is integrated into a single dataset, forming an enhanced multimodal training dataset that contains richer and more diverse data features and better simulates the various changes and interferences found in real-world scenarios. Using this enhanced data for model training improves the robustness and generalization capabilities of the AI ​​security behavior recognition model, enabling it to more accurately identify security behavior events in diverse real-world situations.

[0099] In a non-limiting embodiment, after the security behavior event recognition result of the target monitoring scene is output, the method further includes: performing multimodal confidence verification processing on the security behavior event recognition result to generate a behavior event verification result; calculating the environmental anomaly index in the current time window based on the sound field distribution characteristics and temperature gradient characteristics in the environmental state feature set; inputting the event category of the security behavior event recognition result and the environmental anomaly index into a predefined rule matching engine to calculate the event-environment correlation score; when the correlation score is lower than a preset threshold, performing re-analysis processing of the visual imaging data stream and extracting micro-motion detail features of the target object; performing secondary feature splicing on the micro-motion detail features and the fused monitoring feature vector to generate an enhanced verification feature vector; calling the AI ​​security behavior recognition model to reclassify the enhanced verification feature vector and update the security behavior event recognition result to output the behavior event verification result.

[0100] In actual application scenarios in large commercial plazas, after the AI ​​security behavior recognition model outputs a security behavior event recognition result, it first undergoes a multimodal confidence check. This multimodal confidence check comprehensively considers multiple aspects of information, including visual and environmental information, to assess the credibility of the recognition result. For example, a predefined algorithm calculates a confidence score based on factors such as the model's reliance on various features when generating the recognition result and the consistency between data from different modalities. A high confidence score indicates a reliable recognition result; a low score requires further verification.

[0101] Next, the environmental anomaly index within the current time window is calculated based on the sound field distribution characteristics and temperature gradient characteristics in the environmental state feature set. For the sound field distribution characteristics, the energy distribution of each frequency band is analyzed to see how it differs from that in the normal state. If the energy of a certain frequency band is significantly higher or lower than the normal range, it means that there is an abnormality in the sound field. For example, a sudden and substantial increase in low-frequency energy may mean a large-scale gathering of people or an abnormal low-frequency noise source. For the temperature gradient characteristics, the degree of deviation of the current thermal gradient direction and the temperature difference change rate from the historical normal data is calculated. The two features are then combined, for example, the difference values ​​of the sound field distribution characteristics and the temperature gradient characteristics are combined and calculated according to the weights to obtain the environmental anomaly index.

[0102] The event category and environmental anomaly index of the security behavior event identification results are then fed into a predefined rule-matching engine. The rule-matching engine includes predefined rules for determining the correlation between different event categories and environmental anomaly indices. For example, if the event category is "potential risk behavior category" and the environmental anomaly index exceeds a certain threshold, the correlation is high; conversely, if the environmental anomaly index is low, the correlation is likely low. The rule-matching engine calculates the event-environment correlation score.

[0103] When the correlation score is lower than the preset threshold, it indicates that the current recognition result may be inaccurate and requires further analysis. At this time, the visual imaging data stream is re-analyzed, focusing on extracting the micro-movement detail features of the target object. For example, in the video footage of a commercial plaza, carefully observe the subtle hand movements and slight changes in body posture of the people. These micro-movement details may not have been fully noticed in the previous analysis, but in some cases they may play a key role in the judgment of security behavior events. For example, a person who appears to be walking normally may have an unusual fisting motion, which may indicate potential tension or threatening behavior.

[0104] Perform secondary feature concatenation on the extracted micro-motion detail features and the fused monitoring feature vector. Convert the micro-motion detail features into a suitable feature vector format and then concatenate them with the original fused monitoring feature vector. For example, align the micro-motion detail feature vector with the fused monitoring feature vector along the dimension to form an enhanced verification feature vector. This vector contains richer information and can more comprehensively describe the target monitoring scenario.

[0105] Finally, the AI ​​security behavior recognition model is invoked to reclassify the enhanced verification feature vector. The model analyzes and processes this more informative enhanced verification feature vector and re-outputs the security behavior event recognition results. This updated result, after more detailed analysis and verification, more accurately reflects the actual security behavior events in the target monitoring scenario. The final output, the behavior event verification result, improves recognition accuracy and reliability.

[0106] In a non-limiting embodiment, after the security behavior event recognition result of the output target monitoring scene is output, the method further includes: generating a model error feedback signal based on the classification confidence of the security behavior event recognition result; extracting feature dimensions that have not been fully learned in the current fusion monitoring feature vector to generate a feature sparsity distribution map; adjusting the convolution kernel weight distribution of the AI ​​security behavior recognition model based on the model error feedback signal and the feature sparsity distribution map; performing real-time noise level detection on the environmental sensor data stream and increasing the weight ratio of the motion state feature set in the feature splicing layer when the noise intensity exceeds a set threshold; and synchronizing the adjusted model parameters to the edge computing node to achieve model update.

[0107] During the operation of the surveillance system in a large commercial plaza, after outputting the security behavior event recognition results, a model error feedback signal is first generated based on the classification confidence of the recognition results. Classification confidence reflects the model's degree of certainty in its output results. For example, if the model outputs a confidence level of 0.9 for a certain event category, the model is relatively certain of the result; if the confidence level is 0.5, the model's judgment is somewhat uncertain. By setting a reasonable confidence threshold, when the confidence level of the recognition result falls below this threshold, the model's judgment is considered to be inaccurate, and a model error feedback signal is generated, indicating that the model may need adjustment and optimization in certain aspects.

[0108] Next, the model extracts the under-learned feature dimensions in the current fused monitoring feature vector and generates a feature sparsity distribution map. By analyzing the model's utilization of each feature dimension when processing the fused monitoring feature vector, it determines which feature dimensions are under-mined and under-learned. For example, in the fused monitoring feature vector, the model may not fully capture the correlation between certain feature dimensions related to changes in ambient light intensity and security behavior events, resulting in insufficient utilization of information in these dimensions. These under-learned feature dimensions are presented as a distribution map to intuitively demonstrate the model's deficiencies in feature learning.

[0109] Based on the model error feedback signal and the feature sparsity distribution map, the convolution kernel weight distribution of the AI ​​security behavior recognition model is adjusted. The convolution kernel is used to extract features in the model, and its weight determines the degree of attention paid to different features. According to the model error feedback signal, it is understood in which aspects the model has made judgment errors, and combined with the feature sparsity distribution map, the feature dimensions that need to be strengthened are determined. Then, the weight of the convolution kernel is adjusted in a targeted manner to increase the attention weight on the feature dimensions that have not been fully learned. For example, if it is found that the dimensions related to the high-frequency band characteristics of the sound field intensity have not been fully learned, and the model makes errors in identifying certain security behavior events involving high-frequency sounds, the weight of the convolution kernel responsible for extracting high-frequency band features is appropriately increased so that the model can better capture these feature information in subsequent processing.

[0110] At the same time, the environmental sensor data stream is subjected to real-time noise level detection. In commercial plazas, environmental sensors may be subject to various noise interferences, affecting the accuracy of the data. By real-time monitoring of the noise level in the environmental sensor data stream, for example, noise analysis is performed on the sound field intensity data to determine whether the noise intensity exceeds the set threshold. When the noise intensity exceeds the threshold, it means that the environmental data has been greatly disturbed. At this time, the weight of the motion state feature set in the feature splicing layer is increased. Because when the environmental data is unreliable, it is relatively more dependent on visual motion state features to judge security behavior events. For example, the weight of the motion state feature set in the feature splicing layer is increased from the original 40% to 60% to highlight the importance of motion state features in recognition.

[0111] Finally, the adjusted model parameters are synchronized to the edge computing nodes to implement model updates. In the monitoring system of a large commercial plaza, edge computing nodes are distributed across various monitoring areas and are responsible for real-time data processing and analysis. The model parameters adjusted on the central server are sent to each edge computing node, allowing the models running on these nodes to be updated in a timely manner. This allows the edge computing nodes to use the updated model to identify security behavior events when processing monitoring data later, improving the performance and accuracy of the entire monitoring system and better adapting to the complex and changing real-world environment of the commercial plaza.

[0112] In practical applications, technical details can be improved based on multimodal feature fusion methods (such as ST-GCN spatiotemporal graph convolutional network) and deep learning frameworks (such as TensorFlow's attention mechanism module).

[0113] Specifically, for posture analysis models, the pre-trained model structure of open-source posture estimation frameworks such as OpenPose can be used. Its 18 joint keypoint detection algorithms can be used to accurately identify changes in joint angles, thereby fully exposing the logic for extracting limb extension amplitude features. For spatiotemporal alignment, the Dynamic Time Warping (DTW) algorithm can be combined to calculate nonlinear time offsets between motion trajectories and environmental feature sequences. Sliding window correlation analysis can be used to determine the optimal compensation strategy to avoid feature distortion caused by simple linear interpolation.

[0114] Furthermore, for the spectral decomposition of environmental sensor data, a Mel filter group can be introduced to divide the sound field intensity data into frequency bands that are consistent with the human ear perception characteristics. Combined with the window function parameter setting of the short-time Fourier transform (such as the Hamming window length of 256ms), the process of extracting the sound field distribution characteristics conforms to the common practice in the field of acoustic analysis.

[0115] During model training, the Adam optimizer's adaptive learning rate mechanism, combined with gradient clipping (threshold set to 1.0), and the Focal Loss parameter setting of γ = 2 can be used to automatically balance the weight distribution of scarce categories, effectively addressing the class imbalance problem. When aligning feature dimensions, PCA dimensionality reduction can be used to project features of different dimensions, such as displacement rate (m / s), joint angle (radian), and temperature gradient (°C / m), into a latent space of uniform dimension. Z-score normalization can then be used to eliminate dimensional differences.

[0116] Furthermore, to address environmental noise interference, the signal-to-noise ratio of sound field data can be calculated in real time using the Welch power spectrum estimation method. When a noise level exceeding 45dB is detected, the motion feature weight enhancement mechanism is automatically triggered, and the fusion ratio of the feature splicing layer is dynamically adjusted through gated attention units. Furthermore, the ONNX runtime framework can be used to synchronize parameters of the trained model to edge nodes, and TensorRT's model optimization technology can be used to ensure inference efficiency on edge devices.

[0117] It should be noted that the collection process of the multimodal monitoring data set in the embodiment of the present invention strictly complies with personal information protection laws and regulations and data usage guidelines, and ensures the full knowledge and legal rights of relevant users through a triple authorization mechanism. Before the implementation of the target monitoring scenario, the operator will deploy data collection notification signs in prominent locations at the main entrances and exits of the commercial plaza, service desks and electronic guide screens, and use Chinese and English to scroll the monitoring area range, sensor type and data usage instructions. A dynamic authorization agreement will be embedded in the privacy policy module of the plaza's official APP at the same time, requiring users to complete the data collection informed confirmation operation through their mobile phones when entering the monitoring area for the first time. For visitors who do not have the APP installed, an intelligent interactive terminal will be set up at the entrance of the venue to obtain the explicit authorization of temporary visitors by voice broadcast combined with touch confirmation.

[0118] During the data collection equipment deployment phase, all high-definition cameras were equipped with privacy masking algorithms to mask noise in non-public areas (such as locker room aisles and entrances to nursing rooms). Edge computing nodes were used to remove personally identifiable biometric data in real time. The environmental sensor network employed differential privacy technology, adding random noise perturbations compliant with ISO / IEEE 11073 standards when collecting temperature gradient data, ensuring that individual user surface radiation information could not be reversed. The sound field intensity acquisition system implemented real-time voiceprint desensitization using an embedded DSP processor, performing frequency domain obfuscation on vocal segments exceeding 300ms, in compliance with GB / T 35273-2020 requirements.

[0119] In addition, the system has established a hierarchical access control mechanism. Raw data streams are encrypted and stored using the nationally encrypted SM4 algorithm, and access is restricted to security administrators certified at Level 3 Security Protection using a biometric key. Data retention strictly adheres to relevant regulations. Anonymized behavioral trajectory data is retained for a certain number of days, automatically triggering the blockchain evidence destruction process. Environmental sensor data, after aggregation and desensitization, can be appropriately extended for management purposes. Furthermore, data security impact assessments are regularly conducted by authorized third-party organizations. A dedicated window for exercising data rights has been established in the plaza's service hall, allowing users to query their personal data profiles, submit correction requests, or exercise their right to delete data using their identity credentials.

[0120] The embodiment of the present invention first collects a multimodal monitoring data set of the target monitoring scene, providing a rich source of information for a comprehensive understanding of the monitoring scene; secondly, spatial feature extraction is performed on the visual imaging data stream to accurately obtain the motion state feature set of the target object, which can deeply explore the dynamic situation of the target object in space; then, time series feature extraction is performed on the environmental sensor data stream to obtain an environmental state feature set including sound field distribution, temperature gradient and light intensity, and then characterize the environmental state change from the time dimension; then the motion state feature set and the environmental state feature set are spatially and temporally fused to generate a fused monitoring feature vector, so that different types of features complement each other to form a comprehensive and unique feature representation; finally, the AI ​​security behavior recognition model is called to perform event classification processing on the fused monitoring feature vector, which can fully explore the potential relationship between the data and accurately output the security behavior event recognition results of the target monitoring scene, greatly improving the accuracy and comprehensiveness of security behavior event recognition, and effectively ensuring the safety of the monitoring scene.

[0121] In summary, the embodiments of the present invention innovatively collect multimodal monitoring data, perform targeted feature extraction and spatiotemporal fusion of different data streams, and then use the AI ​​security behavior recognition model to classify events, which can effectively improve the accuracy of security behavior event recognition and overcome the shortcomings of existing technologies.

[0122] An embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for identifying security behavior events based on multimodal analysis is implemented.

[0123] An embodiment of the present invention provides a processor, which is used to run a program, wherein the security behavior event identification method based on multimodal analysis is executed when the program is running.

[0124] In the embodiment of the present invention, Figure 2 As shown, the security behavior event recognition system 100 includes at least one processor 101, and at least one memory 102 and a bus 103 connected to the processor 101; wherein the processor 101 and the memory 102 communicate with each other through the bus 103; the processor 101 is used to call the program instructions in the memory 102 to execute the above-mentioned security behavior event recognition method based on multimodal analysis.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, security behavior event recognition systems (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] In a typical configuration, the security behavior event recognition system includes one or more processors (CPUs), memory, and a bus. The security behavior event recognition system may also include input / output interfaces, network interfaces, and the like.

[0127] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory includes at least one memory chip. Memory is an example of a computer-readable medium.

[0128] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage, computer-readable storage media, or any other non-transmission media that can be used to store information accessible by the security action event recognition system. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or computer-readable storage medium that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or computer-readable storage medium. In the absence of further limitations, an element defined by the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, commodity, or computer-readable storage medium that includes the element.

[0130] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0131] The above are merely embodiments of the present invention and are not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A security behavior event recognition method based on multimodal analysis, characterized in that: include: Collecting a multimodal monitoring data set of a target monitoring scene, wherein the multimodal monitoring data set includes a visual imaging data stream and an environmental sensing data stream; Performing spatial feature extraction processing on the visual imaging data stream to obtain a motion state feature set of the target object; Performing time series feature extraction processing on the environmental sensor data stream to obtain an environmental state feature set, wherein the environmental state feature set includes sound field distribution features, temperature gradient features, and light intensity features; The motion state feature set and the environmental state feature set are subjected to spatiotemporal fusion processing to generate a fused monitoring feature vector, the AI ​​security behavior recognition model is called to perform event classification processing on the fused monitoring feature vector, and the security behavior event recognition result of the target monitoring scene is output.

2. The security behavior event identification method based on multimodal analysis according to claim 1 is characterized in that: The performing spatial feature extraction processing on the visual imaging data stream to obtain a motion state feature set of the target object includes: Performing frame-by-frame parsing on the visual imaging data stream to extract pixel distribution areas of target objects in a continuous frame sequence; Performing contour detection processing on the pixel distribution area to generate a posture contour boundary of the target object; generating a three-dimensional motion trajectory model based on the deformation parameters of the posture contour boundary, and extracting the displacement rate and direction change of each key node in the three-dimensional motion trajectory model; Performing window sliding average processing on the displacement rate and the direction change to generate a smoothed motion trajectory feature; Calling a pre-trained posture analysis model to identify the joint angle changes at the posture contour boundary, and extracting the limb extension amplitude characteristics and movement frequency characteristics of the target object; The motion state feature set is obtained according to the smoothed motion trajectory feature, the limb extension amplitude feature and the action frequency feature.

3. The security behavior event identification method based on multimodal analysis according to claim 2 is characterized in that: The step of performing time series feature extraction processing on the environmental sensor data stream to obtain an environmental state feature set includes: Performing sampling rate alignment processing on the environmental sensor data stream to convert the sound field intensity data, temperature gradient data, and light intensity data of different frequencies into a synchronized data sequence with a unified timestamp; Performing spectrum decomposition processing on the sound field intensity data in the synchronous data sequence, and extracting energy distribution ratio characteristics of each frequency band as the sound field distribution characteristics; Performing spatial interpolation processing on the temperature gradient data to generate a two-dimensional temperature field distribution map, and extracting the thermal gradient direction characteristics and the temperature difference change rate characteristics from the two-dimensional temperature field distribution map as the temperature gradient characteristics; performing abnormal fluctuation detection processing on the light intensity data, and identifying the duration and fluctuation amplitude of the sudden change in light intensity as the light intensity feature; An environmental state feature set is generated based on the sound field distribution feature, the temperature gradient feature, and the light intensity feature.

4. The security behavior event identification method based on multimodal analysis according to claim 3 is characterized in that: The performing spatiotemporal fusion processing on the motion state feature set and the environmental state feature set to generate a fused monitoring feature vector includes: Performing time stamp processing on the smoothed motion trajectory features in the motion state feature set to generate a first time series feature sequence; Performing time stamp processing on the energy distribution ratio feature in the environmental state feature set to generate a second time series feature sequence; Inputting the first time series feature sequence and the second time series feature sequence into a spatiotemporal alignment model to determine a time offset between the smoothed motion trajectory feature and the energy distribution ratio feature; Performing interpolation compensation processing on the first time series feature sequence based on the time offset to generate a compensated motion trajectory feature sequence; The compensated motion trajectory feature sequence, the limb extension amplitude feature, the movement frequency feature, the thermal gradient direction feature, the temperature difference change rate feature, the duration and the fluctuation amplitude are input into the feature splicing layer for dimensional alignment processing to generate a fusion monitoring feature vector.

5. The security behavior event identification method based on multimodal analysis according to claim 4 is characterized in that: The calling of the AI ​​security behavior recognition model to perform event classification processing on the fused monitoring feature vector and outputting the security behavior event recognition result of the target monitoring scene includes: Inputting the fused monitoring feature vector into the feature encoding layer of the AI ​​security behavior recognition model to generate a multi-dimensional abstract feature representation; Performing multi-scale convolution processing on the multi-dimensional abstract feature representation to extract local correlation features and global distribution features under different receptive fields; Inputting the local correlation features and the global distribution features into the attention weight allocation layer to generate feature importance weights; Performing weighted fusion processing on the local correlation features and the global distribution features based on the feature importance weights to generate an optimized feature representation; The optimized feature representation is input into a fully connected classification layer, and the security behavior event recognition result is output. The security behavior event recognition result includes a normal behavior category, a potential risk behavior category, and an emergency threat behavior category.

6. The security behavior event identification method based on multimodal analysis according to claim 5 is characterized in that: The training process of the AI ​​security behavior recognition model includes: Acquire a multimodal training data set of historical monitoring scenes, wherein the multimodal training data set includes a visual training data stream and an environmental training data stream annotated with security behavior event categories; Performing motion state feature extraction processing on the visual training data stream to obtain a training motion state feature set; Performing time series feature extraction processing on the environmental training data stream to obtain a training environmental state feature set; Performing spatiotemporal fusion processing on the training motion state feature set and the training environment state feature set to generate a training fused monitoring feature vector; Obtaining an initial neural network model, inputting the training fusion monitoring feature vector into the initial neural network model for forward propagation processing to obtain a predicted security behavior event category; Calculate the cross entropy loss between the predicted security behavior event category and the labeled security behavior event category, and update the weight parameters of the initial neural network model based on the cross entropy loss until the loss converges to obtain an AI security behavior recognition model.

7. The security behavior event identification method based on multimodal analysis according to claim 6 is characterized in that: The multimodal training data set further includes a spatial training data stream, and the method further includes: Performing random frame sampling processing on the visual training data stream to generate visual training subsets with different time resolutions; Performing noise injection processing on the environmental training data stream to simulate training data under different environmental interference conditions; Performing coordinate perturbation processing on the spatial training data stream to generate training samples with spatial position shift; The visual training subset, the environmental training data stream after the noise injection processing, and the spatial training data stream after the coordinate perturbation processing are merged to generate an enhanced multimodal training data set.

8. The security behavior event identification method based on multimodal analysis according to claim 7 is characterized in that: The calculating the cross entropy loss between the predicted security behavior event category and the labeled security behavior event category includes: Generating a category probability distribution vector according to the predicted security behavior event category; Performing one-hot encoding processing on the labeled security behavior event category to generate a target category probability distribution vector; Calculate the KL divergence between the category probability distribution vector and the target category probability distribution vector to generate an initial loss value; Performing a category weight adjustment process on the initial loss value to increase the loss contribution ratio of the scarce category to obtain an adjusted loss value; Performing smoothing regularization processing on the adjusted loss value to generate the cross entropy loss; The step of updating the weight parameters of the initial neural network model based on the cross entropy loss until a loss convergence state is reached to obtain an AI security behavior recognition model includes: Calculating the gradient of the cross entropy loss with respect to the weight parameters of each layer of the initial neural network model; Performing adaptive learning rate adjustment processing on the gradient value to generate an optimized gradient update direction; Iteratively updating the weight parameters according to the optimized gradient update direction; After each iterative update, the weight parameters are clipped to limit the range of change of the weight parameters; When the loss value of multiple consecutive iterations decreases by less than the preset threshold, it is determined that the loss convergence state has been reached, the training process is terminated, and the current weight parameters are saved to obtain the AI ​​security behavior recognition model.

9. A security behavior event recognition system, characterized in that: It includes a processor and a memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the security behavior event recognition method based on multimodal analysis according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, the method for identifying security behavior events based on multimodal analysis according to any one of claims 1 to 8 is implemented.

Images