Real-time video abnormal behavior detection method based on multi-mode collaborative learning

Abstract

The invention discloses a real-time video abnormal behavior detection method based on multi-mode collaborative learning. The method comprises the following steps: S1, collecting video, audio and environment data through a camera, a microphone and an environment sensor; s2, the collected multi-modal data is preprocessed; s3, an improved multi-modal variational auto-encoder is adopted to generate a multi-modal feature vector; s4, inputting the multi-modal feature vector into a multi-modal diffusion model to generate a multi-modal fusion feature representation; s5, dynamically adjusting feature weight distribution based on a zebra optimization algorithm, and optimizing de-noising parameters and step size setting; s6, outputting an abnormal behavior detection result and a confidence score, and triggering an alarm module to give an alarm in real time; and S7, dynamically adjusting model parameters, and uploading abnormal data to a cloud platform. Through the multi-mode collaborative learning method, high-precision detection and quick response of real-time video abnormal behaviors in a complex dynamic environment are realized, and the safety and stability of an intelligent monitoring system are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a real-time video abnormal behavior detection method based on multimodal collaborative learning. Background Art

[0002] In recent years, with the rapid development of information technologies such as artificial intelligence, big data, and cloud computing, video surveillance systems have been widely used in public safety, traffic management, intelligent security, and other fields. Traditional methods for detecting abnormal video behavior mainly rely on single video data for analysis and judgment. These methods typically use techniques based on motion detection, target tracking, and background modeling to identify abnormal behavior in video streams. However, due to problems such as light changes, occlusions, motion blur, and background interference in video data itself, traditional methods often have difficulty accurately identifying abnormal behavior in practical applications and are prone to false detection or missed detection when dealing with complex scenarios. In addition, the limitations of single-modality data make the detection model insufficiently robust to environmental interference, making it difficult to cope with dynamic and changing actual monitoring scenarios.

[0003] While some existing deep learning-based abnormal behavior detection methods have improved detection accuracy to a certain extent, these methods often require large amounts of labeled data for training and suffer from significant deficiencies in real-time performance and computational complexity. Deep neural network models require enormous computational effort when processing massive amounts of video data, making it difficult to meet the low latency and high response speed requirements of real-time monitoring systems. Furthermore, single-modality detection methods fail to fully utilize information beyond video data, such as audio signals and environmental sensor data. This results in a lack of multi-dimensional, multi-angle information support for the model's identification of abnormal behavior, reducing the comprehensiveness and accuracy of detection.

[0004] In recent years, multimodal collaborative learning technology has gradually become a research hotspot. By fusing information from multiple data modalities such as video, audio, and environment, it can comprehensively analyze monitoring scenarios from different angles and levels, thereby improving the robustness and accuracy of abnormal behavior detection. Despite this, existing multimodal data fusion methods still face many challenges in practical applications. First, there are significant differences in the acquisition, preprocessing, and feature extraction technologies of different modal data. How to achieve effective alignment and collaborative fusion between different modalities remains a difficult problem that needs to be solved urgently. Secondly, due to the inconsistency of the modal data in time and space, traditional fusion methods are prone to introduce noise or redundant information, thereby affecting the final detection results. Furthermore, multimodal collaborative learning models generally have problems with large computational complexity and high response delay in terms of real-time performance, and cannot meet the dual requirements of high real-time performance and high accuracy of actual monitoring systems.

[0005] Therefore, how to provide a real-time video abnormal behavior detection method based on multimodal collaborative learning is a problem that technicians in this field urgently need to solve. Summary of the Invention

[0006] One objective of the present invention is to propose a real-time video abnormal behavior detection method based on multimodal collaborative learning. This method leverages multimodal data fusion, an improved multimodal variational autoencoder, an innovative multimodal diffusion model, and an improved zebra optimization algorithm. Through the technical means of multimodal collaborative learning, the method aims to achieve comprehensive analysis of video, audio, and environmental data and accurate abnormal behavior detection. Compared with existing technologies, this method introduces multiple innovative technologies in the three core aspects of data feature extraction, feature fusion, and model inference, significantly improving the detection accuracy and real-time performance of the system in complex scenarios.

[0007] The method for detecting abnormal behavior in real-time video based on multimodal collaborative learning according to an embodiment of the present invention includes the following steps: S1, collect video data, audio data and environmental data through cameras, microphones and environmental sensors to construct multimodal data; S2. Preprocessing of the collected multimodal data, including image enhancement and normalization of video data, spectral feature extraction of audio data, and normalization and noise filtering of environmental data; S3. Using an improved multimodal variational autoencoder to extract features from the preprocessed video data, audio data, and environmental data to generate a multimodal feature vector; S4, inputting the multimodal feature vector into the multimodal diffusion model, performing feature fusion in the high-noise latent space through the diffusion process, and generating a multimodal fusion feature representation; S5. Based on the zebra optimization algorithm, in the feature fusion process of the multimodal diffusion model, the weight distribution of different modal features is dynamically adjusted to optimize the denoising parameters and step size settings in the multimodal diffusion model; S6. Utilize the optimized multimodal diffusion model for model inference, analyze abnormal behavior based on the multimodal fusion feature representation, output abnormal behavior detection results and confidence scores, and trigger the alarm module to issue a real-time alarm when abnormal behavior is detected. S7. When the camera perspective changes, the scene lighting changes, or the environmental data fluctuates, the detection accuracy and response speed of the multimodal diffusion model are maintained through dynamic parameter adjustment, and the abnormal data is uploaded to the cloud platform for storage.

[0008] Optionally, the S3 specifically includes: S31. Build an improved multimodal variational autoencoder model and set the input channels to correspond to video data, audio data, and environmental data respectively; S32, the pre-processed video data, audio data and environmental data are input into the encoder respectively, mapped to the latent space through the deep neural network, and the corresponding latent variable representations are obtained, which are respectively recorded as 、 and , and define the joint latent variable , where f( ) is the nonlinear fusion function; S33. Calculate the information contribution of each modal latent variable representation: ; in, is the activation value of the j-th dimension feature of modality i, is the characteristic mean of mode i, To prevent the small constant from dividing by zero, N is the feature dimension, v represents the video data, a is the audio data, e is the environment data, Represents the contribution of the feature information of the i-th mode. According to the contribution of the feature information, the features of each mode are adaptively enhanced, the features with high contribution are retained, and the enhanced latent variables are output 、 and ; S34. In the latent space, a cross-modal consistency loss function is introduced to align the enhanced latent variables by calculating the lower bound of the mutual information between latent variables of different modalities: ; Among them, sim( ) represents cosine similarity, is the preset threshold, represents the cross-modal consistency alignment loss, and represents the enhanced latent variable representation of mode i, and max represents the maximum value function; S35, input the aligned enhanced latent variables into the decoder, reconstruct each modal data through the decoding network, and obtain the reconstructed output video data respectively. , audio data and environmental data : ; in, 、 and is the reconstruction weight coefficient of each mode, is the joint reconstruction loss, 、 and Respectively represent the original video data, audio data and environmental data input after preprocessing; S36. Adopt an adaptive feature fusion strategy to perform weighted concatenation on the reconstruction features of each modality to generate the final multimodal feature vector F: ; in, 、 and is the weighting coefficient, is the feature fusion operator.

[0009] Optionally, the S4 specifically includes: S41, using the generated multimodal feature vector F as the initial input of the multimodal diffusion model. The multimodal feature vector serves as the basic data of the entire diffusion process, capturing the key information and semantic features in the multimodal information; S42, performing adaptive noise injection processing on the input multimodal feature vector F: ; in, represents the characteristic representation of the diffusion process at step t, is the dynamic noise term, is the multiplicative scaling factor; S43, under the condition of the initial setting of the diffusion step number T, the forward diffusion process is performed on the multimodal feature vector in the order of t=1,2,…,T, so that the feature vector after each step of noise injection gradually expands from the low-noise state to the high-noise potential space, forming a series of intermediate representations , ,…, ; S44. After the forward diffusion is completed, iterative denoising updates are performed from t=T to t=1, and the gradient information is accelerated by introducing the momentum term: ; in, is the noise estimation function, represents the weight, is the momentum term, is the momentum scaling factor, represents the scaling factor at step t, It represents the feature vector obtained in the t-1th step during the denoising iteration process; S45. In the iterative denoising update process, a multi-stage consistency constraint mechanism is introduced to achieve the coordinated alignment of global and local features by defining a multi-stage consistency loss function: ; in, is the denoising feature vector obtained by the final iteration, It represents the subvector in the cth cluster after the multimodal feature vector F is divided according to the local clustering method. is the mean of all subvectors in the cth cluster, C is the total number of local clusters, and is the weight coefficient, is a multi-stage consistency loss function; S46, the characteristic vector after momentum acceleration and consistency constraint As the final multimodal fusion feature representation ,The multimodal fusion feature representation integrates the global and local feature information of ,video, audio and environmental data in the diffusion and ,denoising process.

[0010] Optionally, the S5 specifically includes: S51, set the initial modal weights of video, audio and environmental data to be 、 and , satisfying the normalization condition; S52. Construct weight optimization objective function , used to quantify multimodal fusion feature representation The difference between the multimodal feature vector F and the weight optimization objective function uses the Euclidean distance metric as the evaluation indicator for weight update: ; in, represents the weight, is the adjustment parameter of the cosine similarity penalty term, is the adjustment parameter of the weighted balance penalty term, v represents video data, a represents audio data, and e represents environmental data; S53, introduce the improved zebra optimization algorithm, in the candidate solution set In which i∈{v,a,e} represents the mode, k represents the candidate solution number, and the adaptive stripe guidance factor is introduced Conduct multi-level collaborative searches: ; in, is the average value of the weight of mode i in the candidate solution set, To adjust the parameters, exp() is an exponential function, and the adaptive fringe guidance factor is used to update the modal weight of the candidate solution: ; in, is the learning rate, is the objective function With respect to the gradient of the weight of mode i in candidate solution k, is the positive attraction factor, is the weight of the best performing mode i in the candidate solution set, is the adjustment coefficient, is the modal i weight of the kth candidate solution in the updated candidate solution; S54. Based on the modal weight update results after the improved zebra optimization algorithm, calculate the standard deviation of each modal weight in the candidate solution set and mean ; And according to the change rate of each modal weight after optimization , dynamically adjust the denoising scaling factor and diffusion steps in the multimodal diffusion model: ; ; in, is the original denoising scaling factor of the multimodal diffusion model at step t, is the updated denoising scaling factor, T is the number of diffusion steps initially set, is the actual number of diffusion steps after adjustment, To prevent division by zero, 、 、 and is the adjustment parameter, ln( ) is the logarithmic function; S55, repeat steps S53 and S54, iteratively update the modal weights in the candidate solution set until the weight optimization objective function When the value drops to the preset threshold or the change is lower than the set tolerance in several consecutive iterations, the optimal modal weight after convergence is finally output. , , and the corresponding updated denoising scaling factor and number of diffusion steps.

[0011] Optionally, the S6 specifically includes: S61. Representing multimodal fusion features The data is input to the multimodal diffusion model inference module, which uses the feature information after iterative denoising and reconstruction during the diffusion process to perform model inference. S62. In the multimodal diffusion model inference module, an iterative denoising inference process is used to represent the input multimodal fusion features and calculate the abnormal behavior detection score. : ; Among them, w is the weight vector of the inference module, b is the bias term, express The information entropy of is the entropy adjustment factor, is a high-order interaction feature, is the high-order feature adjustment parameter, exp() is the exponential function, and ln() is the logarithmic function; S63. Based on abnormal behavior detection score And multimodal fusion feature representation The feature dispersion of , calculate the confidence score C: ; in, represents the characteristic variance, is the variance adjustment parameter, is the interaction complexity index extracted during the inference process of the multimodal diffusion model, To adjust the parameters; S64. Abnormal behavior detection score Compare with the preset threshold to determine the abnormal behavior: ; in, Preset thresholds for abnormal behavior detection; S65. When the judgment result is that abnormal behavior exists, output alarm signal A: ; Among them, A=1 means triggering an alarm; S66, comprehensive output abnormal behavior detection score , confidence score C and alarm signal A, constitute the detection result vector .

[0012] The beneficial effects of the present invention are: The present invention significantly improves the accuracy, real-time performance and adaptability of abnormal behavior detection through a real-time video abnormal behavior detection method based on multimodal collaborative learning, overcoming many deficiencies in the prior art. In terms of multimodal data fusion, the present invention uses an improved multimodal variational autoencoder for feature extraction, combining the multi-dimensional information of video, audio and environmental data to achieve deep fusion and effective alignment of different modal features. The adaptive noise injection and momentum-accelerated iterative denoising update technology introduced in the diffusion model enable multimodal features to maintain the consistency and robustness of feature fusion in high-noise latent spaces, avoiding the problem of susceptibility to noise interference in traditional methods.

[0013] Furthermore, this invention introduces an improved zebra optimization algorithm to dynamically adjust the weight distribution of each modal feature during the feature fusion process. This algorithm proposes an adaptive stripe guidance factor and a multi-level collaborative search strategy, enabling the model to optimize feature weights in real time based on actual scene changes, thereby achieving adaptive capabilities in dynamic and complex environments. This innovation not only enhances the model's refined control over the contribution of multimodal features but also incorporates the influencing factors of information entropy and feature distribution dispersion into the weight optimization process, effectively improving the accuracy of feature fusion and the stability of model inference.

[0014] During the model inference stage, the present invention uses an innovative abnormal behavior detection score calculation method and a multimodal diffusion model for model inference, introducing information entropy and high-order interaction characteristics into the inference formula. This allows the abnormal behavior detection score to reflect the degree of abnormal behavior while sensitively capturing high-order interaction information in the features. In addition, the confidence score calculation formula not only relies on the detection score, but also combines multimodal feature variance and complexity indicators, further enhancing the credibility of the detection results. In practical applications, the detection results of the present invention contain three dimensional information: abnormal detection score, confidence score, and alarm signal, providing richer and more accurate data support for security monitoring systems.

[0015] The present invention has a strong real-time response capability. When the camera viewing angle changes, the scene lighting changes, or the environmental data fluctuates, the detection accuracy and response speed of the multimodal diffusion model are always maintained through a dynamic parameter adjustment mechanism. Especially when responding to emergencies and urgent situations, the real-time and accuracy of the present invention can effectively reduce security risks and enhance the prevention and control capabilities in the public safety field. Through the automated abnormal behavior detection and alarm linkage mechanism, the intelligent monitoring system has achieved a transformation from "passive monitoring" to "active prevention and control", which has broad application prospects and significant social benefits in many fields such as intelligent transportation, smart cities, enterprise security, and social public safety.

[0016] In summary, the present invention not only overcomes the problems of poor multimodal data fusion, insufficient abnormal behavior detection accuracy, and poor real-time performance in the existing technology, but also achieves accurate abnormal behavior detection and real-time response in complex dynamic environments through innovative algorithm design and model inference methods, providing an efficient, reliable and scalable technical solution in the fields of intelligent monitoring and public safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1This is a flow chart of the real-time video abnormal behavior detection method based on multimodal collaborative learning proposed by the present invention; Figure 2 This is a schematic diagram of the model reasoning process of the multimodal diffusion model of the real-time video abnormal behavior detection method based on multimodal collaborative learning proposed in the present invention. DETAILED DESCRIPTION

[0018] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0019] refer to Figure 1 and Figure 2 ,A real-time video abnormal behavior detection method based on multimodal collaborative learning,,including the following steps: S1, collect video data, audio data and environmental data through cameras, microphones and environmental sensors to construct multimodal data; S2. Preprocessing of the collected multimodal data, including image enhancement and normalization of video data, spectral feature extraction of audio data, and normalization and noise filtering of environmental data; S3. Using an improved multimodal variational autoencoder to extract features from the preprocessed video data, audio data, and environmental data to generate a multimodal feature vector; S4, inputting the multimodal feature vector into the multimodal diffusion model, performing feature fusion in the high-noise latent space through the diffusion process, and generating a multimodal fusion feature representation; S5. Based on the zebra optimization algorithm, in the feature fusion process of the multimodal diffusion model, the weight distribution of different modal features is dynamically adjusted to optimize the denoising parameters and step size settings in the multimodal diffusion model; S6. Utilize the optimized multimodal diffusion model for model inference, analyze abnormal behavior based on the multimodal fusion feature representation, output abnormal behavior detection results and confidence scores, and trigger the alarm module to issue a real-time alarm when abnormal behavior is detected. S7. When the camera perspective changes, the scene lighting changes, or the environmental data fluctuates, the detection accuracy and response speed of the multimodal diffusion model are maintained through dynamic parameter adjustment, and the abnormal data is uploaded to the cloud platform for storage.

[0020] In this embodiment, S3 specifically includes: S31. Build an improved multimodal variational autoencoder model and set the input channels to correspond to video data, audio data, and environmental data respectively; S32, the pre-processed video data, audio data and environmental data are input into the encoder respectively, mapped to the latent space through the deep neural network, and the corresponding latent variable representations are obtained, which are respectively recorded as 、 and , and define the joint latent variable , where f( ) is the nonlinear fusion function; S33. Calculate the information contribution of each modal latent variable representation: ; in, is the activation value of the j-th dimension feature of modality i, is the characteristic mean of mode i, To prevent the small constant from dividing by zero, N is the feature dimension, v represents the video data, a is the audio data, e is the environment data, Represents the contribution of the feature information of the i-th mode. According to the contribution of the feature information, the features of each mode are adaptively enhanced, the features with high contribution are retained, and the enhanced latent variables are output 、 and ; S34. In the latent space, a cross-modal consistency loss function is introduced to align the enhanced latent variables by calculating the lower bound of the mutual information between latent variables of different modalities: ; Among them, sim( ) represents cosine similarity, is the preset threshold, represents the cross-modal consistency alignment loss, and represents the enhanced latent variable representation of mode i, and max represents the maximum value function; S35, input the aligned enhanced latent variables into the decoder, reconstruct each modal data through the decoding network, and obtain the reconstructed output video data respectively. , audio data and environmental data : ; in, 、 and is the reconstruction weight coefficient of each mode, is the joint reconstruction loss, 、 and Respectively represent the original video data, audio data and environmental data input after preprocessing; S36. Adopt an adaptive feature fusion strategy to perform weighted concatenation on the reconstruction features of each modality to generate the final multimodal feature vector F: ; in, 、 and is the weighting coefficient, is the feature fusion operator.

[0021] In this embodiment, the S4 specifically includes: S41, using the generated multimodal feature vector F as the initial input of the multimodal diffusion model. The multimodal feature vector serves as the basic data of the entire diffusion process, capturing the key information and semantic features in the multimodal information; S42, performing adaptive noise injection processing on the input multimodal feature vector F: ; in, represents the characteristic representation of the diffusion process at step t, is the dynamic noise term, is the multiplicative scaling factor; S43, under the condition of the initial setting of the diffusion step number T, the forward diffusion process is performed on the multimodal feature vector in the order of t=1,2,…,T, so that the feature vector after each step of noise injection gradually expands from the low-noise state to the high-noise potential space, forming a series of intermediate representations , ,…, ; S44. After the forward diffusion is completed, iterative denoising updates are performed from t=T to t=1, and the gradient information is accelerated by introducing the momentum term: ; in, is the noise estimation function, represents the weight, is the momentum term, is the momentum scaling factor, represents the scaling factor at step t, It represents the feature vector obtained in the t-1th step during the denoising iteration process; S45. In the iterative denoising update process, a multi-stage consistency constraint mechanism is introduced to achieve the coordinated alignment of global and local features by defining a multi-stage consistency loss function: ; in, is the denoising feature vector obtained by the final iteration, It represents the subvector in the cth cluster after the multimodal feature vector F is divided according to the local clustering method. is the mean of all subvectors in the cth cluster, C is the total number of local clusters, and is the weight coefficient, is a multi-stage consistency loss function; S46, the characteristic vector after momentum acceleration and consistency constraint As the final multimodal fusion feature representation ,The multimodal fusion feature representation integrates the global and local feature information of ,video, audio and environmental data in the diffusion and ,denoising process.

[0022] In this embodiment, the S5 specifically includes: S51, set the initial modal weights of video, audio and environmental data to be 、 and , satisfying the normalization condition; S52. Construct weight optimization objective function , used to quantify multimodal fusion feature representation The difference between the multimodal feature vector F and the weight optimization objective function uses the Euclidean distance metric as the evaluation indicator for weight update: ; in, represents the weight, is the adjustment parameter of the cosine similarity penalty term, is the adjustment parameter of the weighted balance penalty term, v represents video data, a represents audio data, and e represents environmental data; S53, introduce the improved zebra optimization algorithm, in the candidate solution set In which i∈{v,a,e} represents the mode, k represents the candidate solution number, and the adaptive stripe guidance factor is introduced Conduct multi-level collaborative searches: ; in, is the average value of the weight of mode i in the candidate solution set, To adjust the parameters, exp() is an exponential function, and the adaptive fringe guidance factor is used to update the modal weight of the candidate solution: ; in, is the learning rate, is the objective function With respect to the gradient of the weight of mode i in candidate solution k, is the positive attraction factor, is the weight of the best performing mode i in the candidate solution set, is the adjustment coefficient, is the modal i weight of the kth candidate solution in the updated candidate solution; S54. Based on the modal weight update results after the improved zebra optimization algorithm, calculate the standard deviation of each modal weight in the candidate solution set and mean ; And according to the change rate of each modal weight after optimization , dynamically adjust the denoising scaling factor and diffusion steps in the multimodal diffusion model: ; ; in, is the original denoising scaling factor of the multimodal diffusion model at step t, is the updated denoising scaling factor, T is the number of diffusion steps initially set, is the actual number of diffusion steps after adjustment, To prevent division by zero, 、 、 and is the adjustment parameter, ln( ) is the logarithmic function; S55, repeat steps S53 and S54, iteratively update the modal weights in the candidate solution set until the weight optimization objective function When the value drops to the preset threshold or the change is lower than the set tolerance in several consecutive iterations, the optimal modal weight after convergence is finally output. , , and the corresponding updated denoising scaling factor and number of diffusion steps.

[0023] In this embodiment, S6 specifically includes: S61. Representing multimodal fusion features The data is input to the multimodal diffusion model inference module, which uses the feature information after iterative denoising and reconstruction during the diffusion process to perform model inference. S62. In the multimodal diffusion model inference module, an iterative denoising inference process is used to represent the input multimodal fusion features and calculate the abnormal behavior detection score. : ; Among them, w is the weight vector of the inference module, b is the bias term, express The information entropy of is the entropy adjustment factor, is a high-order interaction feature, is the high-order feature adjustment parameter, exp() is the exponential function, and ln() is the logarithmic function; S63. Based on abnormal behavior detection score And multimodal fusion feature representation The feature dispersion of , calculate the confidence score C: ; in, represents the characteristic variance, is the variance adjustment parameter, is the interaction complexity index extracted during the inference process of the multimodal diffusion model, To adjust the parameters; S64. Abnormal behavior detection score Compare with the preset threshold to determine the abnormal behavior: ; in, Preset thresholds for abnormal behavior detection; S65. When the judgment result is that abnormal behavior exists, output alarm signal A: ; Among them, A=1 means triggering an alarm; S66, comprehensive output abnormal behavior detection score , confidence score C and alarm signal A, constitute the detection result vector .

[0024] Example 1: To verify the feasibility of this invention, it was applied to the intelligent campus security system of a key middle school in a certain city. The campus covers an area of ​​approximately 60,000 square meters and includes multiple functional areas, including teaching, living, and exercise areas. The system is equipped with 150 high-definition cameras, 40 environmental sensors (such as temperature, humidity, smoke, and light sensors), and multiple microphones for multimodal acquisition of video, audio, and environmental data. By integrating with existing monitoring systems and emergency warning platforms, the intelligent campus security system enables real-time detection and rapid response to abnormal behavior on campus.

[0025] In actual application scenarios, the system of the present invention is mainly used to monitor crowded areas during peak hours of classes, evening self-study periods, and large-scale campus activities. In the corridors of teaching buildings, when the system detects abnormal behaviors such as students running, pushing, and falling through video, combined with high-decibel screams in the audio signal and a sharp increase in crowd density in the environmental sensor data, the system can quickly determine that there may be safety hazards and immediately push alarm information to the mobile devices of security personnel. In public places such as campus cafeterias and playgrounds, when it is detected that the crowd density exceeds the safety threshold, the system automatically issues an early warning to remind management personnel to conduct timely guidance, thereby effectively preventing congestion and stampede accidents.

[0026] The present invention extracts key features from video, audio, and environmental data through an improved multimodal variational autoencoder, achieves deep feature fusion through a multimodal diffusion model, and combines information entropy and high-order interaction characteristics in the model reasoning process, thereby improving the accuracy of abnormal behavior detection scores. During a campus emergency drill, the system detected an abnormal increase in smoke concentration through environmental sensors. At the same time, the video captured the rapid gathering of the crowd to the exit and the alarm sound was recognized in the audio. It took only 1.2 seconds to complete the abnormal behavior detection and push fire warning information to the management personnel. Compared with the traditional single video monitoring method, the present invention significantly shortens the detection response time and effectively improves the detection accuracy through the collaborative analysis of multimodal data.

[0027] In order to verify the actual effect of the present invention, the system's abnormal behavior detection data was recorded in detail during a three-month test, and compared and analyzed with the traditional single-modal video surveillance method. The evaluation indicators mainly include detection accuracy, false alarm rate, response time, and the stability of the system in complex scenarios. In terms of detection accuracy, the present invention reached 97.2%, while the traditional method was 86.5%; in terms of false alarm rate, the present invention was only 2.8%, while the traditional method was as high as 9.1%; in terms of average response time, the response time of the present invention was 1.4 seconds, while the traditional method required 3.7 seconds; under the conditions of changes in camera viewing angle, light changes, and environmental data fluctuations, the detection accuracy fluctuation of the system of the present invention was controlled within ±2.3%, while the detection accuracy fluctuation of the traditional method exceeded ±7.5%.

[0028] In specific application scenarios, such as monitoring in the corridors of teaching buildings, the present invention can accurately identify more than 90% of abnormal behavior incidents, and issue warnings 3-5 seconds in advance when students are crowded or pushing each other, effectively avoiding multiple potential safety accidents. In the campus cafeteria, when it is detected that the flow of people reaches 90% of the safe capacity, the system automatically starts the "crowd control mode" and cooperates with the audio broadcast system to divert diners in time, significantly improving order during peak hours. In sports venues, by analyzing the movement trajectories of the crowd in the video, the noise characteristics in the audio, and the data fluctuations of the environmental sensors, the present invention effectively identified multiple student conflict incidents and notified security personnel to arrive on the scene to deal with them within 5 seconds through campus broadcasts and mobile devices.

[0029] Table 1 Performance comparison between multimodal collaborative learning method and traditional method in smart campus security ; As can be seen from the data in the table, the present invention's real-time video abnormal behavior detection method based on multimodal collaborative learning demonstrates significant advantages over traditional methods across multiple key performance indicators. In terms of detection accuracy, the present invention achieved 97.2%, a 10.7 percentage point improvement over the traditional method's 86.5%. This improvement is primarily due to multimodal data fusion technology and innovative model inference methods, enabling the system to achieve higher-precision abnormal behavior identification in a variety of scenarios.

[0030] The false alarm rate is a key indicator of the reliability of an anomaly detection system. The proposed method achieved a false alarm rate of only 2.8%, compared to 9.1% for traditional methods. This 6.3 percentage point reduction in false alarm rate means that the system can more accurately identify actual abnormal behavior in real-world applications, effectively reducing unnecessary interventions and alarms. This performance is particularly critical in public security monitoring scenarios, as it can avoid wasted resources and management confusion caused by false alarms.

[0031] In terms of response speed, the method of the present invention has an average response time of only 1.4 seconds, far lower than the 3.7 seconds of traditional methods. This shortened response time directly improves the system's emergency response capabilities to abnormal events. In practical applications, it can trigger alarms and linkage emergency mechanisms more quickly, especially in campus scenarios where student safety is a concern, significantly reducing the risk of accidents.

[0032] The method also demonstrates excellent stability in complex dynamic scenes. Under conditions of camera viewing angle changes, lighting variations, and environmental data fluctuations, the system's detection accuracy fluctuates by only ±2.3%, compared to a range of up to ±7.5% for traditional methods. This demonstrates that the multimodal diffusion model and dynamic weight adjustment algorithm of the present invention can effectively adapt to complex scene changes, maintaining the robustness and reliability of the detection model.

[0033] In terms of the specific number of abnormal events identified, the inventive method identified 560 abnormal events during the testing period, while the traditional method only identified 485, an increase of 75 events. This significant improvement is due to the inventive method's deep analysis of multimodal data and efficient feature extraction capabilities, which enables the system to mine more abnormal behavior events in the same data environment, helping managers to promptly address potential risks.

[0034] Finally, the proposed method achieved a real-time alarm accuracy rate of 96.8%, significantly higher than the 84.2% achieved by traditional methods. This demonstrates that in actual alarm triggering scenarios, nearly all alarms were based on actual abnormal behavior, effectively avoiding false positives and missed positives, and improving the system's response efficiency and decision-making support capabilities in emergencies.

[0035] In summary, the technical solution of the present invention significantly outperforms traditional methods in many aspects, including detection accuracy, response speed, system stability, and alarm accuracy. It demonstrates particularly high practical application value in the diverse and complex scenarios of campus security monitoring. These data demonstrate the significant advantages of the present invention in improving the safety, reliability, and emergency response capabilities of smart campus security systems, and demonstrate its broad market application prospects and social benefits.

[0036] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A real-time video abnormal behavior detection method based on multimodal collaborative learning, characterized by: The steps include: S1, collect video data, audio data and environmental data through cameras, microphones and environmental sensors to construct multimodal data; S2. Preprocessing of the collected multimodal data, including image enhancement and normalization of video data, spectral feature extraction of audio data, and normalization and noise filtering of environmental data; S3. Using an improved multimodal variational autoencoder to extract features from the preprocessed video data, audio data, and environmental data to generate a multimodal feature vector; S4, inputting the multimodal feature vector into the multimodal diffusion model, performing feature fusion in the high-noise latent space through the diffusion process, and generating a multimodal fusion feature representation; S5. Based on the zebra optimization algorithm, in the feature fusion process of the multimodal diffusion model, the weight distribution of different modal features is dynamically adjusted to optimize the denoising parameters and step size settings in the multimodal diffusion model; S6. Utilize the optimized multimodal diffusion model for model inference, analyze abnormal behavior based on the multimodal fusion feature representation, output abnormal behavior detection results and confidence scores, and trigger the alarm module to issue a real-time alarm when abnormal behavior is detected. S7. When the camera perspective changes, the scene lighting changes, or the environmental data fluctuates, the detection accuracy and response speed of the multimodal diffusion model are maintained through dynamic parameter adjustment, and the abnormal data is uploaded to the cloud platform for storage.

2. The method for detecting abnormal behavior in real-time video based on multimodal collaborative learning according to claim 1 is characterized in that: The S3 specifically includes: S31. Build an improved multimodal variational autoencoder model and set the input channels to correspond to video data, audio data, and environmental data respectively; S32, the pre-processed video data, audio data and environmental data are input into the encoder respectively, mapped to the latent space through the deep neural network, and the corresponding latent variable representations are obtained, which are respectively recorded as 、 and , and define the joint latent variable , where f( ) is the nonlinear fusion function; S33. Calculate the information contribution of each modal latent variable representation: ； in, is the activation value of the j-th dimension feature of modality i, is the characteristic mean of mode i, To prevent the small constant from dividing by zero, N is the feature dimension, v represents the video data, a is the audio data, e is the environment data, Represents the contribution of the feature information of the i-th mode. According to the contribution of the feature information, the features of each mode are adaptively enhanced, the features with high contribution are retained, and the enhanced latent variables are output 、 and ; S34. In the latent space, a cross-modal consistency loss function is introduced to align the enhanced latent variables by calculating the lower bound of the mutual information between latent variables of different modalities: ； Among them, sim( ) represents cosine similarity, is the preset threshold, represents the cross-modal consistency alignment loss, and represents the enhanced latent variable representation of mode i, and max represents the maximum value function; S35, input the aligned enhanced latent variables into the decoder, reconstruct each modal data through the decoding network, and obtain the reconstructed output video data respectively. , audio data and environmental data : ； in, 、 and is the reconstruction weight coefficient of each mode, is the joint reconstruction loss, 、 and Respectively represent the original video data, audio data and environmental data input after preprocessing; S36. Adopt an adaptive feature fusion strategy to perform weighted concatenation on the reconstruction features of each modality to generate the final multimodal feature vector F: ; in, 、 and is the weighting coefficient, is the feature fusion operator.

3. The real-time video abnormal behavior detection method based on multimodal collaborative learning according to claim 1 is characterized in that: The S4 specifically includes: S41, using the generated multimodal feature vector F as the initial input of the multimodal diffusion model. The multimodal feature vector serves as the basic data of the entire diffusion process, capturing the key information and semantic features in the multimodal information; S42. Adaptive noise injection processing is performed on the input multimodal feature vector F: ; in, represents the characteristic representation of the diffusion process at step t, is the dynamic noise term, is the multiplicative scaling factor; S43, under the condition of the initial setting of the diffusion step number T, the forward diffusion process is performed on the multimodal feature vector in the order of t=1,2,…,T, so that the feature vector after each step of noise injection gradually expands from the low-noise state to the high-noise potential space, forming a series of intermediate representations , ,…, ; S44. After the forward diffusion is completed, iterative denoising updates are performed from t=T to t=1, and the gradient information is accelerated by introducing the momentum term: ; in, is the noise estimation function, represents the weight, is the momentum term, is the momentum scaling factor, represents the scaling factor at step t, It represents the feature vector obtained in the t-1th step during the denoising iteration process; S45. In the iterative denoising update process, a multi-stage consistency constraint mechanism is introduced to achieve the coordinated alignment of global and local features by defining a multi-stage consistency loss function: ; in, is the denoising feature vector obtained by the final iteration, It represents the subvector in the cth cluster after the multimodal feature vector F is divided according to the local clustering method. is the mean of all subvectors in the cth cluster, C is the total number of local clusters, and is the weight coefficient, is a multi-stage consistency loss function; S46, the characteristic vector after momentum acceleration and consistency constraint As the final multimodal fusion feature representation ,The multimodal fusion feature representation integrates the global and local feature information of ,video, audio and environmental data in the diffusion and ,denoising process.

4. The method for detecting abnormal behavior in real-time video based on multimodal collaborative learning according to claim 1, characterized in that: The S5 specifically includes: S51, set the initial modal weights of video, audio and environmental data to be 、 and , satisfying the normalization condition; S52. Construct weight optimization objective function , used to quantify multimodal fusion feature representation The difference between the multimodal feature vector F and the weight optimization objective function uses the Euclidean distance metric as the evaluation indicator for weight update: ; in, represents the weight, is the adjustment parameter of the cosine similarity penalty term, is the adjustment parameter of the weighted balance penalty term, v represents video data, a represents audio data, and e represents environmental data; S53, introduce the improved zebra optimization algorithm, in the candidate solution set In which i∈{v,a,e} represents the mode, k represents the candidate solution number, and the adaptive stripe guidance factor is introduced Conduct multi-level collaborative searches: ; in, is the average value of the weight of mode i in the candidate solution set, To adjust the parameters, exp() is an exponential function, and the adaptive fringe guidance factor is used to update the modal weight of the candidate solution: ; in, is the learning rate, is the objective function With respect to the gradient of the weight of mode i in candidate solution k, is the positive attraction factor, is the weight of the best performing mode i in the candidate solution set, is the adjustment coefficient, is the modal i weight of the kth candidate solution in the updated candidate solution; S54. Based on the modal weight update results after the improved zebra optimization algorithm, calculate the standard deviation of each modal weight in the candidate solution set and mean ; And according to the change rate of each modal weight after optimization , dynamically adjust the denoising scaling factor and diffusion steps in the multimodal diffusion model: ; ; in, is the original denoising scaling factor of the multimodal diffusion model at step t, is the updated denoising scaling factor, T is the number of diffusion steps initially set, is the actual number of diffusion steps after adjustment, To prevent division by zero, 、 、 and is the adjustment parameter, ln( ) is the logarithmic function; S55, repeat steps S53 and S54, iteratively update the modal weights in the candidate solution set until the weight optimization objective function When the value drops to the preset threshold or the change is lower than the set tolerance in several consecutive iterations, the optimal modal weight after convergence is finally output. , , and the corresponding updated denoising scaling factor and number of diffusion steps.

5. The method for detecting abnormal behavior in real-time video based on multimodal collaborative learning according to claim 1, characterized in that: The S6 specifically includes: S61. Representing multimodal fusion features The data is input to the multimodal diffusion model inference module, which uses the feature information after iterative denoising and reconstruction during the diffusion process to perform model inference. S62. In the multimodal diffusion model inference module, an iterative denoising inference process is used to represent the input multimodal fusion features and calculate the abnormal behavior detection score. : ； Among them, w is the weight vector of the inference module, b is the bias term, express The information entropy of is the entropy adjustment factor, is a high-order interaction feature, is the high-order feature adjustment parameter, exp() is the exponential function, and ln() is the logarithmic function; S63. Based on abnormal behavior detection score And multimodal fusion feature representation The feature dispersion of , calculate the confidence score C: ; in, represents the characteristic variance, is the variance adjustment parameter, is the interaction complexity index extracted during the inference process of the multimodal diffusion model, To adjust the parameters; S64. Abnormal behavior detection score Compare with the preset threshold to determine the abnormal behavior: ; in, Preset thresholds for abnormal behavior detection; S65. When the judgment result is that abnormal behavior exists, output alarm signal A: ; Among them, A=1 means triggering an alarm; S66, comprehensive output abnormal behavior detection score , confidence score C and alarm signal A, constitute the detection result vector .

Images