A multi-modal home security anomaly detection method, device and medium

Abstract

This invention provides a multimodal home security anomaly detection method, device, and medium. The method includes: acquiring multimodal home monitoring data; extracting feature vectors from each modality of the multimodal home monitoring data and concatenating and fusing them with candidate anomaly semantic vectors to obtain a multimodal feature tensor; inputting the multimodal feature tensor into a trained lightweight neural network model to output the confidence weight of each modality; optimizing the confidence weight of each modality through DDPG reinforcement learning to obtain the optimized confidence weight of each modality; and inputting the confidence weights of all modalities into a pre-trained Transformer sequence modeling network to obtain the recognition result of home security anomaly detection. This method, device, and medium can solve the problem that existing home security anomaly detection methods mostly rely on a single modality for anomaly detection, resulting in low recognition accuracy in complex environments.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology, and in particular to a multimodal home security anomaly detection method, device and medium. Background Technology

[0002] With the rapid development of smart homes and the Internet of Things, home monitoring systems are gradually evolving towards intelligence and multimodal integration. Existing monitoring systems mostly rely on a single modality (such as video) for anomaly detection, but they suffer from low accuracy in complex environments. For example, insufficient lighting, obstructions, and environmental noise severely affect the perception effect of a single video modality, making it difficult to comprehensively capture the diverse information in abnormal behavior. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a multimodal home security anomaly detection method, device and medium, so as to solve the problem that the existing home security anomaly detection methods mostly rely on a single mode for anomaly detection, resulting in low recognition accuracy in complex environments.

[0004] In a first aspect, the present invention provides a multimodal home security anomaly detection method, comprising:

[0005] Acquire multimodal data from home surveillance, wherein the multimodal data from home surveillance includes: video data, audio data, and infrared data;

[0006] Multiple modality mapping features are extracted from the home surveillance multimodal data, and each modality mapping feature is concatenated and fused with a candidate anomaly semantic vector to obtain a multimodal feature tensor, wherein the candidate anomaly semantic vector is generated based on a pre-trained text model;

[0007] The multimodal feature tensor is input into a trained lightweight neural network model. The lightweight neural network model evaluates the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics, and outputs the confidence weight of each modality. The trained lightweight neural network model is trained based on the semantic alignment loss function.

[0008] The confidence weights of each modality are optimized by DDPG reinforcement learning to obtain the optimized confidence weights of each modality, wherein the DDPG reinforcement learning uses detection accuracy, confidence variance and feedback consistency as reward functions;

[0009] Based on the optimized confidence weights of all the modalities and the mapping features of all the modalities, the identification results of home security anomaly detection are obtained through a pre-trained Transformer sequence modeling network.

[0010] Furthermore, before acquiring the home monitoring multimodal data, the method further includes:

[0011] Define multiple exception event labels;

[0012] By calling the pre-trained text model, each abnormal event label is converted into an abnormal semantic vector, thus forming an abnormal semantic vector representation library.

[0013] Furthermore, the acquisition of multimodal data from home monitoring specifically includes:

[0014] The video data, audio data, and infrared data are collected, and the time domain of the video data, audio data, and infrared data is synchronized based on a unified timestamp mechanism.

[0015] Further, the step of extracting multiple modality mapping features from the home surveillance multimodal data and concatenating and fusing each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor specifically includes:

[0016] The video data is input into a pre-trained ResNet network to extract video feature vectors;

[0017] The audio data is input into the VGGish audio encoder to extract the audio feature vector;

[0018] Infrared data is input into a dedicated convolutional encoder to extract infrared feature vectors;

[0019] Based on the video feature vector, audio feature vector, and infrared feature vector, each modality mapping feature is obtained through a modality-independent fully connected layer, wherein all the modality mapping features have the same dimension;

[0020] Each modality mapping feature is concatenated with the current candidate anomaly semantic vector in the anomaly semantic vector representation library, and the concatenated modality mapping features are stacked to obtain the multimodal feature tensor.

[0021] Furthermore, before inputting the multimodal feature tensor into a trained lightweight neural network model, evaluating the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics through the lightweight neural network model, and outputting the confidence weight of each modality, the method further includes:

[0022] Obtain the confidence weights of multiple historical modalities from the output of the historical multimodal feature tensor of the lightweight neural network model, and calculate the cosine similarity between the mapped features of each historical modality and the corresponding anomaly semantic vector.

[0023] A semantic alignment loss function is defined based on the cosine similarity between each historical modality mapping feature and the corresponding abnormal semantic vector;

[0024] The lightweight neural network model is trained based on the semantic alignment loss function and the classification cross-entropy loss function corresponding to the confidence weights of all the historical modalities, resulting in the trained lightweight neural network model.

[0025] Furthermore, before optimizing the confidence weights of each modality using DDPG reinforcement learning to obtain the optimized confidence weights of each modality, the method further includes:

[0026] The detection accuracy, confidence variance, and feedback consistency are designed as reward functions.

[0027] Based on the reward function, the DDPG reinforcement learning algorithm is used to learn the optimal strategy for optimizing the confidence weights of each modality.

[0028] Furthermore, after obtaining the identification result of home security anomaly detection through a pre-trained Transformer sequence modeling network based on the optimized confidence weights of all the modalities and all the modal mapping features, the method further includes:

[0029] Based on the semantic closed-loop learning process built through user interaction feedback mechanism, the abnormal semantic vector representation library dynamically expanded through incremental learning, and the confidence weight of each modality after introducing Beta distribution for historical modeling and Bayesian update, continuous learning and recognition of abnormal types and device status are achieved.

[0030] Secondly, the present invention provides a multimodal home security anomaly detection device, comprising:

[0031] The acquisition module is used to acquire multimodal data from home surveillance, wherein the multimodal data from home surveillance includes: video data, audio data, and infrared data;

[0032] An extraction and splicing module, connected to the acquisition module, is used to extract multiple modality mapping features from the home monitoring multimodal data, and splice and fuse each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor, wherein the candidate anomaly semantic vector is generated based on a pre-trained text model;

[0033] An input evaluation module, connected to the extraction and splicing module, is used to input the multimodal feature tensor into a trained lightweight neural network model. The lightweight neural network model evaluates the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics, and outputs the confidence weight of each modality. The trained lightweight neural network model is trained based on a semantic alignment loss function.

[0034] An optimization module, connected to the input evaluation module, is used to optimize the confidence weight of each modality through DDPG reinforcement learning to obtain the optimized confidence weight of each modality, wherein the DDPG reinforcement learning uses detection accuracy, confidence variance and feedback consistency as reward functions;

[0035] The module is connected to the optimization module and is used to obtain the identification result of home security anomaly detection through a pre-trained Transformer sequence modeling network based on the confidence weights of all the optimized modalities and the mapping features of all the modalities.

[0036] Thirdly, the present invention provides a multimodal home security anomaly detection device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the multimodal home security anomaly detection method described in the first aspect above.

[0037] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal home security anomaly detection method described in the first aspect.

[0038] The multimodal home security anomaly detection method, device, and medium provided by this invention first acquire multimodal home monitoring data, including video data, audio data, and infrared data. Then, multiple modality mapping features are extracted from the multimodal home monitoring data, and each modality mapping feature is concatenated and fused with a candidate anomaly semantic vector to obtain a multimodal feature tensor. The candidate anomaly semantic vector is generated based on a pre-trained text model. Next, the multimodal feature tensor is input into a trained lightweight neural network model, which then analyzes the modalities and candidate anomaly semantic vectors within the multimodal feature tensor. The degree of matching of abnormal semantics is evaluated, and the confidence weight of each modality is output. The trained lightweight neural network model is trained based on the semantic alignment loss function. Then, the confidence weight of each modality is optimized through DDPG reinforcement learning to obtain the optimized confidence weight of each modality. The DDPG reinforcement learning uses detection accuracy, confidence variance, and feedback consistency as reward functions. Finally, based on the optimized confidence weights of all modalities and the mapping features of all modalities, the identification result of home security anomaly detection is obtained through a pre-trained Transformer sequence modeling network. This invention differs from methods that rely solely on single-modality information for anomaly detection by acquiring multimodal data from home surveillance. It introduces anomaly semantic vectors generated from a pre-trained text model as guiding information, which are then concatenated and fused with multimodal mapping features. This achieves deep coupling between anomaly semantics and multimodal perception features, effectively improving the ability to discriminate specific anomaly types. Through a semantic alignment loss function, a lightweight neural network model is trained to estimate the matching confidence of each modality with candidate anomaly semantics, achieving semantically guided modality selection optimization. This overcomes the adaptability bottleneck of traditional fixed or attention-based fusion methods. Furthermore, the DDPG reinforcement learning framework, with detection accuracy, confidence variance, and feedback consistency as reward functions, further optimizes the confidence weights of modalities, enabling adaptive scheduling strategies across scenarios and anomaly categories. This effectively improves robustness and flexibility in complex and changing environments, significantly increasing the accuracy of anomaly detection in complex environments. This solves the problem that existing home security anomaly detection methods often rely on single-modality detection, leading to low accuracy in complex environments. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a multimodal home security anomaly detection method according to Embodiment 1 of the present invention;

[0041] Figure 2 This is a flowchart of another multimodal home security anomaly detection method according to Embodiment 1 of the present invention;

[0042] Figure 3 This is a flowchart of the semantic-confidence collaboration mechanism of the multimodal home security anomaly detection method based on semantic vector feedback driven according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of a multimodal home security anomaly detection device according to Embodiment 2 of the present invention;

[0044] Figure 5 This is a schematic diagram of the structure of a multimodal home security anomaly detection device according to Embodiment 3 of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0046] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0047] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0048] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0049] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0050] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0051] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0052] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0053] Application Overview

[0054] To improve the robustness of anomaly detection, multimodal data such as audio, infrared, and temperature have been widely introduced for fusion processing in recent years. However, most multimodal fusion methods adopt static fusion strategies and lack dynamic evaluation mechanisms for the information quality and reliability of different modalities in the current scenario. This results in the system being unable to intelligently identify and prioritize the use of key modal information for different anomaly events. Especially when the environment and event types are highly variable, the fusion effect is limited, affecting the detection accuracy and stability.

[0055] Furthermore, existing technologies typically only output a simple judgment of whether an anomaly exists, lacking a deep semantic understanding of abnormal events and making it difficult to achieve dynamic adjustment of modal weights based on anomaly types. The lack of a semantically driven fusion scheduling mechanism makes the system ill-suited to adapting to diverse and complex abnormal behaviors such as intrusions, broken window sounds, and falls, limiting its intelligence level and practical application effectiveness. Meanwhile, the heterogeneity and temporal complexity of multimodal data present a significant challenge: how to achieve efficient and adaptive feature fusion while ensuring temporal synchronization and maintaining the accuracy of feature representation and fusion efficiency.

[0056] More importantly, traditional anomaly detection systems generally lack user interaction and feedback mechanisms, failing to continuously correct and optimize model performance through multi-round semantic interaction. This results in a persistently high false alarm rate, impacting user experience and system reliability. Therefore, there is an urgent need for a dynamic modality weight adjustment method driven by anomaly event semantic vectors, combined with multi-round human-computer interaction semantic feedback, to support continuous system learning and automatic identification of new anomaly event types. The identified new anomaly events not only cover various intrusions and abnormal behaviors but also include damage or malfunctions of sensor devices such as cameras, microphones, and infrared sensors, avoiding missed anomaly events due to device malfunctions. This comprehensively improves the accuracy, robustness, and breadth and depth of anomaly detection, significantly enhancing the intelligence level of home monitoring systems.

[0057] To address the aforementioned technical issues, this application provides a multimodal home security anomaly detection method, device, and medium. By acquiring multimodal home monitoring data, it differs from methods relying solely on single-modal information for anomaly detection. It introduces anomaly semantic vectors generated from a pre-trained text model as guiding information, which are then concatenated and fused with multimodal mapping features. This achieves deep coupling between anomaly semantics and multimodal perception features, effectively enhancing the ability to discriminate specific anomaly types. Through a semantic alignment loss function, a lightweight neural network model is trained to estimate the matching confidence of each modality with candidate anomaly semantics, achieving semantically guided modality selection optimization. This overcomes the adaptability bottleneck of traditional fixed or attention-based fusion methods. Furthermore, by employing the DDPG reinforcement learning framework, with detection accuracy, confidence variance, and feedback consistency as reward functions, the confidence weights of the modalities are further optimized, enabling adaptive scheduling strategies across scenarios and anomaly categories. This effectively improves robustness and flexibility in complex and changing environments, significantly increasing the accuracy of anomaly detection in complex environments. This addresses the problem that existing home security anomaly detection methods often rely on single-modality detection, leading to low accuracy in complex environments.

[0058] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0059] Example 1:

[0060] This embodiment provides a multimodal home security anomaly detection method, such as... Figure 1 As shown, the method includes:

[0061] Step S101: Acquire home monitoring multimodal data, wherein the home monitoring multimodal data includes: video data, audio data, and infrared data.

[0062] It should be noted that the video data is preferably video data from a camera, the audio data is preferably audio data from a microphone, and the infrared data is preferably infrared data from an infrared thermal imaging device.

[0063] In an optional embodiment, before acquiring the home monitoring multimodal data, the method further includes:

[0064] Define multiple exception event labels;

[0065] By calling the pre-trained text model, each abnormal event label is converted into an abnormal semantic vector, thus forming an abnormal semantic vector representation library.

[0066] Specifically, the system-defined abnormal event labels are transformed into fixed-dimensional semantic vectors by calling the pre-trained text encoder Sentence-BERT, thus forming an abnormal type semantic representation library (i.e., an abnormal semantic vector representation library).

[0067] It should be noted that during system operation, new exception types added by user feedback can be used to generate new semantic vectors through the same encoder and expand the semantic library, thereby achieving dynamic updates of exception semantic vectors.

[0068] In one optional embodiment, acquiring home monitoring multimodal data specifically includes:

[0069] The video data, audio data, and infrared data are collected, and the time domain of the video data, audio data, and infrared data is synchronized based on a unified timestamp mechanism.

[0070] Specifically, the system acquires trimodal data from cameras, microphones, and infrared thermal imaging devices in parallel, and synchronizes the multimodal data in the time domain based on a unified timestamp mechanism.

[0071] Specifically, the acquisition and processing strategies are dynamically adjusted based on the semantic labels of candidate anomalies: audio data is extracted using Short Time Fourier Transform (STFT) to obtain time-frequency features, video data is extracted by interval frame extraction to obtain keyframes, and infrared modal data is repaired by linear interpolation to restore lost frame data, all unified into fixed-length, synchronized temporal feature blocks. The system also records the modal data stream status to identify potentially missing or failed modalities.

[0072] Step S102: Extract multiple modality mapping features from the home monitoring multimodal data, and concatenate and fuse each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor, wherein the candidate anomaly semantic vector is generated based on a pre-trained text model.

[0073] In this embodiment, anomaly semantic vectors generated based on a pre-trained text model are introduced for the first time. The acquisition and processing strategies are dynamically adjusted according to the candidate anomaly semantic labels. These vectors are used as guiding information and spliced ​​and fused with multimodal mapping features to achieve deep coupling between the semantics of the detection target and the modal perception features. This effectively improves the system's ability to distinguish specific anomaly types, which is different from the traditional static modeling method based only on visual or modal information.

[0074] In one optional embodiment, the step of extracting multiple modality mapping features from the home surveillance multimodal data and concatenating and fusing each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor specifically includes:

[0075] The video data is input into a pre-trained ResNet network to extract video feature vectors;

[0076] The audio data is input into the VGGish audio encoder to extract the audio feature vector;

[0077] Infrared data is input into a dedicated convolutional encoder to extract infrared feature vectors;

[0078] Based on the video feature vector, audio feature vector, and infrared feature vector, each modality mapping feature is obtained through a modality-independent fully connected layer, wherein all the modality mapping features have the same dimension;

[0079] Each modality mapping feature is concatenated with the current candidate anomaly semantic vector in the anomaly semantic vector representation library, and the concatenated modality mapping features are stacked to obtain the multimodal feature tensor.

[0080] Specifically, the preprocessed video data, audio data, and infrared data are input into the corresponding deep feature extraction networks ResNet, VGGish, and infrared convolutional encoder to generate video feature vectors, audio feature vectors, and infrared feature vectors, respectively. These vectors are then mapped to a shared-dimensional feature space through a fully connected projection layer to obtain each modality mapping feature. Subsequently, each modality mapping feature is fused with the current candidate anomaly semantic vector in the anomaly semantic vector representation library by concatenation. Finally, a semantically enhanced multimodal feature tensor is constructed by stacking and concatenating each modality mapping feature.

[0081] Step S103: Input the multimodal feature tensor into the trained lightweight neural network model, evaluate the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics through the lightweight neural network model, and output the confidence weight of each modality. The trained lightweight neural network model is trained based on the semantic alignment loss function.

[0082] Specifically, the modal features after semantic fusion (i.e., multimodal feature tensors) are input into a lightweight neural network model to quantitatively evaluate the degree of matching between each modality and the current abnormal semantics, and output the confidence weight of each modality.

[0083] In this embodiment, an alignment loss function based on semantic direction consistency is proposed to train a lightweight network to estimate the matching confidence of each modality with the current semantic target, thereby realizing a differentiable modality weight allocation mechanism. This achieves semantically guided modality selection optimization, overcoming the adaptive bottleneck of traditional fixed or attention-based fusion methods.

[0084] In an optional embodiment, before inputting the multimodal feature tensor into a trained lightweight neural network model, evaluating the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics through the lightweight neural network model, and outputting the confidence weight of each modality, the method further includes:

[0085] Obtain the confidence weights of multiple historical modalities from the output of the historical multimodal feature tensor of the lightweight neural network model, and calculate the cosine similarity between the mapped features of each historical modality and the corresponding anomaly semantic vector.

[0086] A semantic alignment loss function is defined based on the cosine similarity between each historical modality mapping feature and the corresponding abnormal semantic vector;

[0087] The lightweight neural network model is trained based on the semantic alignment loss function and the classification cross-entropy loss function corresponding to the confidence weights of all the historical modalities, resulting in the trained lightweight neural network model.

[0088] Specifically, the training process of the lightweight neural network model is as follows: A lightweight modality confidence learning network is designed, which includes two fully connected layers and a non-linear activation function ReLU. The network is used to input multimodal feature tensors and output three modality confidence weight vectors. The network output is first passed through a softmax layer to ensure that the weights are non-negative and sum to 1. Cosine similarity is calculated between each modality feature and semantic vector, and an alignment loss is defined based on the cosine similarity. Modality-semantic dependencies are jointly optimized and learned. During overall training, the lightweight neural network model is trained by jointly using classification cross-entropy loss and alignment loss to obtain the trained lightweight neural network model.

[0089] It is worth mentioning that a semantic alignment loss function is introduced during the training process to optimize the directional consistency between modal features and target semantic vectors, enabling the model to learn the dependency between modal contribution and event semantics, and realize a differentiable confidence allocation mechanism.

[0090] Step S104: Optimize the confidence weight of each modality through DDPG reinforcement learning to obtain the optimized confidence weight of each modality, wherein the DDPG reinforcement learning uses detection accuracy, confidence variance and feedback consistency as reward functions.

[0091] In this embodiment, a reward function is designed with detection accuracy, confidence variance, and feedback consistency as its core. The DDPG reinforcement learning framework is used to continuously optimize the modality fusion weights, thereby realizing an adaptive scheduling strategy across scenarios and anomaly categories, which effectively improves the robustness and flexibility of the system in complex and ever-changing environments.

[0092] In an optional embodiment, before optimizing the confidence weights of each modality using DDPG reinforcement learning to obtain optimized confidence weights for each modality, the method further includes:

[0093] The detection accuracy, confidence variance, and feedback consistency are designed as reward functions.

[0094] Based on the reward function, the DDPG reinforcement learning algorithm is used to learn the optimal strategy for optimizing the confidence weights of each modality.

[0095] Specifically, a reward function based on detection accuracy, confidence variance, and user feedback consistency is constructed. The DDPG reinforcement learning algorithm is introduced to continuously optimize the modality fusion strategy (i.e., optimize the confidence weight of each modality). Under different anomaly types and scenario conditions, the dynamic weighting of each modality feature during fusion is intelligently adjusted, and an environment-adaptive confidence scheduling strategy is learned.

[0096] Step S105: Based on the optimized confidence weights of all the modalities and the mapping features of all the modalities, obtain the identification results of home security anomaly detection through the pre-trained Transformer sequence modeling network.

[0097] Specifically, the confidence weights of all modalities and the mapping features of all modalities after fusion and optimization (weighted summation) are collected, and the fused feature sequences of the most recent L time steps are collected. The fused feature sequences are then input into a pre-trained Transformer sequence modeling network, which performs behavior modeling and abnormal event type classification based on a multi-head self-attention mechanism, and outputs the abnormal classification probability (i.e. the identification result of home security anomaly detection).

[0098] It should be noted that, in order to support the continuous identification of new types of abnormal events, an incremental learning strategy can be introduced to fine-tune the model parameters without interfering with the performance of old type identification, so as to ensure that the system has the ability to continuously learn new types of events during deployment.

[0099] In an optional embodiment, after obtaining the identification result of home security anomaly detection through a pre-trained Transformer sequence modeling network based on the optimized confidence weights of all the modalities and all the modal mapping features, the method further includes:

[0100] Based on the semantic closed-loop learning process built through user interaction feedback mechanism, the abnormal semantic vector representation library dynamically expanded through incremental learning, and the confidence weight of each modality after introducing Beta distribution for historical modeling and Bayesian update, continuous learning and recognition of abnormal types and device status are achieved.

[0101] In this embodiment, a semantic closed-loop learning process is constructed by combining user interaction feedback mechanism. The abnormal semantic library (i.e., the abnormal semantic vector representation library) is dynamically expanded through incremental learning. The Beta distribution is introduced to perform historical modeling and Bayesian updates of modality confidence, so as to realize continuous self-learning and robust identification of abnormal types and device status, and significantly enhance the system evolution capability.

[0102] Specifically, using modality confidence scores as observations, a Beta distribution is employed to probabilistically model the stability of each modality. By comparing with user feedback or prediction accuracy, the Bayesian distribution parameters are updated in real time to enhance the historical dependence of the modality confidence model. When a modality is detected to have prolonged low confidence or abnormal fluctuations, the device anomaly identification mechanism is triggered, automatically reducing the weight of that modality or temporarily blocking it to ensure the reliability of the overall judgment.

[0103] Specifically, after anomaly detection is completed, the system initiates a semantic-level confirmation dialogue with the user and generates a new anomaly annotation record based on the user's selection. This annotation is used to update the anomaly semantic vector library, provide feedback to correct confidence weights or fusion strategies, and participate in the incremental training of subsequent models. Ultimately, this constructs a closed-loop learning mechanism of semantic feedback and model optimization, enhancing the system's adaptability and generalization ability.

[0104] It is worth mentioning that the multimodal home security anomaly detection method that integrates semantic feedback and human-computer interaction learning provided by this invention not only integrates multimodal data, but also realizes modal scheduling based on semantic hierarchy, making the detection system more intelligent, flexible and adaptable to different scenarios, and providing strong technical support for smart home security.

[0105] In a specific embodiment, such as Figure 2 As shown, this multimodal home security anomaly detection method may specifically include:

[0106] Step 1: Anomaly Semantic Vector Initialization and Update: A pre-trained text encoder is used to transform the labels of each anomaly event type into dense semantic vectors, building an initial semantic library. During operation, users can add new anomaly types, and the semantic representation set is dynamically expanded through the same encoder, ensuring that the system has semantic learning and adaptation capabilities.

[0107] Specifically, step 1 includes:

[0108] Step 1.1: Predefine a set of common abnormal event categories in the system, such as falls, intrusions, and broken window sounds, to form a tag set:

[0109] ;

[0110] in, Let k be the set of tags, and k be the number of tags. A brief text description of the k-th anomalous event. This set of tags ensures that the system covers the main anomaly types to be monitored.

[0111] Step 1.2: When the system starts, it loads the pre-trained Sentence-BERT model and performs semantic vector encoding on each label text in the label set. The encoding result is a fixed-dimensional semantic vector.

[0112] ;

[0113] in, A brief text description (i.e., a set of tags) representing the i-th abnormal event. The text of the i-th exception event label in the table is a brief natural language description. This represents the encoding function of a pre-trained text encoder, used to map the input labeled text to a continuous vector space; The semantic vectorization result of label i represents the dense semantic embedding of the label, which is used for subsequent semantic matching, alignment and reasoning; d is the semantic vector dimension with a value of 512. Represents a real vector space of dimension d, used to constrain semantic vectors. The range of values ​​for , i.e. It is a d-dimensional real vector.

[0114] Specifically, in a home security monitoring system, abnormal event categories are predefined during system initialization, including: intrusion, falls, sound of breaking windows, and loud cries for help. After Sentence-BERT encoding, these categories result in four corresponding 512-dimensional vectors. , ,in The semantic vector for "intrusion" For the semantic vector of "falling down", For the semantic vector of "sound of a broken window", This is the semantic vector for "intense cries for help".

[0115] Step 1.3: Construct an anomaly semantic library from all the obtained initial label vectors. This library is used for semantic alignment and matching in subsequent anomaly detection.

[0116] Step 1.4: During system operation, if a user discovers that certain potential anomaly categories are not covered, the user can submit new anomaly text tags through the UI interface. The system then uses the same Sentence-BERT encoder to vectorize the tag, obtaining a new 512-dimensional semantic vector. .

[0117] Step 1.5: Add the newly obtained semantic vector to the original semantic library. The system loads the updated semantic library in real time to ensure that the model can identify the latest anomalies.

[0118] Step 2: Multimodal Data Preprocessing and Time Synchronization: Video, audio, and infrared modal data are acquired in parallel, and time-domain synchronization is achieved using a unified timestamp mechanism. STFT is used to extract time-frequency features from audio, keyframes are extracted from video, and missing frames are repaired using interpolation in infrared images. All data are processed into synchronized, fixed-length temporal feature blocks.

[0119] Specifically, step 2 includes:

[0120] Step 2.1: The system synchronously starts the camera, microphone, and infrared thermal imaging device to collect video, audio, and infrared data streams.

[0121] Specifically, the camera captures images of indoor people's activities, the microphone captures ambient sounds such as voices and the sound of breaking glass, and the infrared thermal imaging device acquires indoor temperature distribution. If the candidate label is "breakable window sound," the audio modality uses a smaller STFT window to improve the resolution of the instantaneous cracking sound. If the candidate label is "fall," the video modality uses higher frame rate sampling to capture details of human movement.

[0122] Step 2.2: Design the system's internal clock, assigning a uniform timestamp 't' to each acquired data frame, and ensuring timing consistency based on the hardware clock and the NTP time synchronization protocol. For example, at t=10s, video frames, audio frames, and infrared images all have a uniform timestamp to ensure subsequent synchronized processing.

[0123] Step 2.3: Perform a Short-Time Fourier Transform (STFT) on the audio signal. The specific calculation formula is as follows:

[0124] ;

[0125] in, This represents the original audio signal segment captured at time t; It is a short-time Fourier transform operator used to extract the spectral features of a signal as it changes over time; This represents the spectral energy at frequency f, which is a frequency domain component of the two-dimensional time-frequency feature map extracted by STFT; f represents the frequency index, with a value range of 1≤f≤F, where F is the total frequency resolution; This represents the complete set of spectral features of the audio at time t, used for subsequent anomaly detection.

[0126] Step 2.4: To reduce computational load, keyframes are extracted from the video signal based on a fixed time interval t. The specific calculation formula is as follows:

[0127] ;

[0128] in, This represents a keyframe in a video after frame extraction. This represents a video frame extracted at time t; This indicates the frame sampling interval, used to control the video frame sampling frequency; It is a non-negative integer index, representing the sampling sequence number; It represents the set of natural numbers.

[0129] Step 2.5: Infrared acquisition often results in frame loss due to equipment limitations. Linear interpolation is used to reconstruct the missing frames. The specific calculation formula is as follows:

[0130] ;

[0131] in, This represents the infrared image frame at time t after reconstruction; , This indicates the two consecutive infrared frames that have been captured. This represents the temporal interpolation operator, which is a linear interpolation used to fill in missing frames; t is the interpolation time, satisfying... .

[0132] Step 2.6: Trim or pad the processed modal data to a uniform length L to generate synchronized multimodal feature blocks, which facilitates subsequent input of time series models.

[0133] Step 2.7, Define Indicators This is used to monitor the data quality of each modality in real time, where 1 represents that modality m is valid at time t, and 0 represents missing or abnormal data. This state is used as a constraint when performing weighted fusion.

[0134] Specifically, in the "broken window sound" scenario, the audio modality =1 and resolution is enhanced; video modality suffers from insufficient nighttime lighting. =0; Infrared mode =1, which can detect the heat source trajectory of a figure entering after the glass breaks.

[0135] Step 3: Modal Feature Mapping and Semantic Fusion: Map each modal feature to a shared dimension through independent fully connected layers, and concatenate it with the current candidate anomaly semantic vector. This forms a semantically guided modal feature tensor, providing a unified and semantically enhanced feature representation for subsequent confidence assessment and behavior modeling.

[0136] Specifically, step 3 includes:

[0137] Step 3.1: Input the video keyframe sequence into the pre-trained ResNet network to extract deep features. = .in, This represents a keyframe in the video at time t; It is a residual network feature extractor; The feature vector of the video modality; It is a video feature dimension, such as 2048.

[0138] Step 3.2: Input the STFT audio result into the VGGish audio encoder to extract the audio feature vector. = .in, The audio STFT spectral characteristics obtained in step 2.3; This represents the VGGish audio coding model; This is the audio modal feature vector; It is an audio feature dimension, such as 128.

[0139] Step 3.3: Input the reconstructed infrared sequence into a dedicated convolutional encoder to obtain feature vectors. = .in, It is the infrared thermal imaging frame after time t has been interpolated and completed; It is a convolution-based infrared image encoder; Infrared modal feature vectors; Indicates the dimension of the infrared feature, such as 512.

[0140] Specifically, in the fall scene, the video modal frame skipping interval is 0.05 seconds. Highlight the contours of the human body as it transitions from upright to fallen, including details such as the angle between the torso and the ground, and the relative positions of the limbs. In a scene with the sound of a breaking window, Highlighting the instantaneous high-energy peak characteristics in the 1kHz~8kHz frequency band, corresponding to the sound spectrum of glass shattering. In intrusion scenarios, Highlight the temperature difference between the human body heat source (36~38℃) and the environment (20~25℃), as well as the continuous movement trajectory of the heat source.

[0141] Step 3.4: Through fully connected layers that are independent for each modality, the mapping feature dimensions are unified to... The specific calculation formula is as follows:

[0142] .

[0143] Where m represents each mode, v represents the video mode, a represents the audio mode, and r represents the infrared mode; These are the linear mapping weight matrices for the corresponding modes, with a unified dimension. ; This refers to the common semantic dimension after mapping features from different modalities. Dimensions for different modal features; This represents the bias vector for the corresponding mode; These are modal feature vectors (i.e., video feature vectors, audio feature vectors, and infrared feature vectors). This represents the modal feature vector (i.e., the mapping features of each modality) after being mapped to the common semantic space.

[0144] Step 3.5: Concatenate the obtained modality mapping features with the current anomaly candidate semantic vector. The specific calculation formula is as follows:

[0145] .

[0146] in, It is a representation after splicing the various modalities; This represents a vector concatenation operation; This represents the semantic vector of the current anomaly candidate label i (i.e., the semantic vectorization result of label i). These are the features of each modality mapping.

[0147] Step 3.6: Stack the concatenated three-modal features in the modal dimension to form a tensor. ,in This represents the concatenation of video, audio, and infrared features with the current anomaly semantic vector. It is a multimodal joint representation.

[0148] Step 4: Semantic-guided multimodal alignment and confidence estimation: A lightweight neural network is used to evaluate the matching degree between each modal feature and the target semantic vector, and outputs the confidence weight. During training, a semantic alignment loss function is introduced to optimize the consistency between modal features and semantic vectors in spatial direction, achieving interpretable modal contribution modeling.

[0149] Specifically, step 4 includes:

[0150] Step 4.1: Design a lightweight modal confidence learning network The network consists of two fully connected layers and a non-linear activation function ReLU, used as input to the multimodal feature tensor. It outputs the three-way modal confidence weight vector:

[0151] ;

[0152] in, It is the initial three-modal confidence weight vector; The confidence weight vectors corresponding to video, audio, and infrared modes.

[0153] Step 4.2: The network output first passes through a softmax layer to ensure that the weights are non-negative and sum to 1. The specific calculation formula is as follows:

[0154] ;

[0155] in, This represents the modal confidence weight of the corresponding mode after normalization; It represents the initial modal confidence weights for the corresponding modalities.

[0156] Step 4.3: Calculate the cosine similarity between each modality feature and the semantic vector. The specific calculation formula is as follows:

[0157] ;

[0158] in, This represents the similarity score between the m-th modality and the anomalous semantic vector. It is the feature vector of the m-th mode (i.e., the feature mapping of each mode). For the anomaly semantic vector (i.e., the semantic vector of the current anomaly candidate label i), the alignment loss is defined as:

[0159] ;

[0160] in, Let L be the semantic alignment loss, and L be the number of time steps.

[0161] Step 4.4: Jointly optimize the learning modality-semantic dependencies, using joint classification cross-entropy loss during overall training. The specific formula for calculating the alignment loss is as follows:

[0162] ;

[0163] in, It is the overall training loss of the model; For classification cross-entropy loss; The weighting coefficients are used to control the importance of semantic alignment loss.

[0164] Specifically, in the "falling" scenario, the video keyframes clearly capture human movement, and the network may output w=[0.65, 0.20, 0.15], indicating that the system relies more on video features. In the "broken window sound" scenario, audio modality features are particularly crucial, and after softmax normalization, we get: w=[0.25, 0.60, 0.15]. In the "intrusion" scenario, infrared thermal imaging accurately captures nighttime movement, with high cosine similarity. The cosine similarity of each modality is as follows: With weights w=[0.30, 0.10, 0.60], the system primarily relies on infrared and video to identify anomalies. When training "fall" category samples, cross-entropy ensures correct model classification, while alignment loss promotes a high degree of matching between video features and "fall" semantic vectors. Audio and infrared features are only used as auxiliary features. In this way, the system dynamically learns the contributions of different modalities under different anomaly scenarios.

[0165] Step 5: Dynamic Confidence Scheduling Based on Reinforcement Learning: The DDPG reinforcement learning algorithm is used to construct a reward function based on detection accuracy, modality confidence variance, and user feedback consistency. The algorithm intelligently learns weighted strategies for each modality under different abnormal scenarios, achieving dynamic optimization and adaptive adjustment of the fusion method.

[0166] Specifically, step 5 includes:

[0167] Step 5.1, the state definition includes the current modality confidence weight vector, historical accuracy, and user feedback consistency index. .in, This represents the system state at time t during reinforcement learning; Output the confidence weight vector for the current three-modal softmax; This is the accuracy rate of the previous testing cycle; It is the user feedback consistency score.

[0168] Step 5.2, the action is defined as the policy vector for adjusting the modal weights. .in, The weight adjustment action at time t; This represents the updated weights for mode m.

[0169] Step 5.3, Overall Detection Accuracy Weighted variance (That is, confidence variance) to prevent extreme dependence on a single modality, combined with consistency with user feedback. The reward function is designed, and the specific calculation formula is as follows:

[0170] ;

[0171] in, This represents the reward value obtained by reinforcement learning at time t; This is the current detection accuracy rate; This indicates consistency in user feedback; 1 represents complete consistency in user confirmation. It is the weighted variance, used to penalize extreme single-modal dependencies; the larger the value, the greater the penalty. , , For hyperparameter weights.

[0172] Step 5.4: Using the DDPG algorithm, input the current state through the Actor network. Output continuous actions Learn the optimal strategy to maximize long-term rewards; then set the state... and current action Input to the Critic network output Q-value estimate Q( , Evaluate state-action pairs, utilize the experience replay buffer, sample training, and continuously update parameters.

[0173] Specifically, in the "fall" detection, the system's previous step weights were [0.65, 0.20, 0.15], with an accuracy of 92%. If the user confirms the prediction is correct, the status is: In a "broken window sound" scenario, if the audio contribution is insufficient, the action may be... =[-0.05, +0.08, -0.03], the adjusted weights become [0.20, 0.68, 0.12]. In nighttime "intrusion" detection, the infrared modality is crucial. If the system gives a prediction accuracy of 88%, and user feedback confirms it's correct, but the infrared weight is too high, leading to excessive variance: =0.7 0.88 + 0.2 1-0.1 0.22 = 0.796, the system receives a positive reward, but the variance penalty encourages a reduction in extreme reliance on infrared in the future. In the "fall" scenario, reinforcement learning maintains a high weight for video and optimizes the auxiliary modality; in the "broken window sound" scenario, reinforcement learning gradually increases the contribution of audio, improving detection accuracy; in the "intrusion" scenario, reinforcement learning learns a dynamic scheduling strategy of "video-dominated during the day and infrared-dominated at night".

[0174] Step 6: Anomaly Classification and Incremental Learning: The fused feature sequence is input into the pre-trained Transformer model, which uses a self-attention mechanism to model behavior and output anomaly categories. Through the incremental learning mechanism, the system can gradually adapt to and learn new anomaly events without forgetting old categories.

[0175] Specifically, step 6 includes:

[0176] Step 6.1: Combine all modal features and perform a weighted sum. The specific calculation formula is as follows:

[0177] ;

[0178] in, It is the fused multimodal integrated feature; Let m be the eigenvector of the m-th mode; The m-th modality weight is dynamically assigned for reinforcement learning (i.e., the confidence weight of each modality after optimization).

[0179] Step 6.2: Collect the most recent L-timestep fused feature sequences. Input a multi-head self-attention Transformer model for temporal modeling, and output the anomaly classification probability. .

[0180] Step 6.3: Receive new anomaly category samples, fine-tune the model parameters, and simultaneously maintain regularization constraints on the old category parameters to prevent forgetting. The specific calculation formula for the incremental learning strategy is as follows:

[0181] ;

[0182] in, Incremental learning loss; The classification loss for the new category; These are the current model parameters; A snapshot of the old category parameters; This is the regularization coefficient.

[0183] Specifically, in the "fall" event, the weights are [0.65, 0.20, 0.15], corresponding to the fused features. =0.65 +0.20 +0.15 At this point, video features dominate the detection, with audio and infrared as auxiliary features. In the "intrusion" event, the system combines the captured infrared and video abnormal motion trajectories from L=10 frames. After modeling the temporal relationship using Transformer, the output prediction results are P(y=intrusion)=0.87, P(y=fall)=0.06, P(y=break window sound)=0.02, and P(y=screaming for help)=0.05. The system determines the event as "intrusion". The initial anomaly label library is {intrusion, fall, break window sound, screaming for help}. Users upload "fire" samples as new anomaly labels. The system uses Sentence-BERT encoding to generate new semantic vectors and adds them to the semantic library. During incremental training of the model, the discriminative ability of the old categories is maintained while learning the "fire" features. After training, the system outputs: P(y=fire)=0.91, P(y=intrusion)=0.01, P(y=fall)=0.05, P(y=sound of breaking window)=0.01, P(y=screaming for help)=0.2. The system successfully identified the new anomaly category, while the performance of the old category did not decrease significantly.

[0184] Step 7: Dynamic Bayesian Update of Confidence Parameters and Equipment Anomaly Identification: A Beta distribution is used to model the correct / incorrect statistics of the confidence level for each modality, and the distribution parameters are dynamically updated based on feedback. The system then identifies modalities with long-term abnormal fluctuations, triggering the equipment anomaly handling mechanism and improving the robustness of overall decision-making.

[0185] Specifically, step 7 includes:

[0186] Step 7.1, assume that the confidence scores for each modality follow a Beta distribution. .

[0187] Step 7.2: Adjust the parameters based on whether the confidence level corresponds to the correct prediction. The specific calculation formula is as follows:

[0188] ;

[0189] in, , These are the initial shape parameters of the Beta distribution; , The updated shape parameters of the Beta distribution; "It is the Beta distribution parameter update learning rate that controls the parameter update speed and avoids drastic fluctuations in the distribution due to a single prediction result." This is an indicator function used to statistically determine whether a prediction is correct or incorrect.

[0190] Step 7.3, when the modal confidence mean is observed to be consistently below the threshold When the variance increases abnormally, anomaly detection is triggered, the device malfunctions, and the weight of that modality is automatically reduced or temporarily disabled. The specific calculation formula is as follows:

[0191] ;

[0192] in, Indicates an anomaly flag under the current modal weights; The confidence level is the mean. The confidence level variance; , It is the threshold for triggering an exception.

[0193] Specifically, in the "fall" event, the video modality prediction was correct. Increased audio noise and prediction errors. Increase. During nighttime intrusions, the infrared confidence distribution remains stable with a mean of [value missing]. =0.6, keep it. The video performs poorly in low light conditions, the mean drops to 0.45, and the variance increases. The system automatically reduces the video weight, and the adjusted weight is [0.20, 0.15, 0.65], with infrared as the dominant detection method.

[0194] Step 8: Multi-round User Interaction Feedback Closed-Loop Optimization: After detection, the system requests user confirmation through semantic dialogue and transforms the feedback results into new training samples and semantic vectors. The semantic library, confidence weights, and fusion strategies are updated to build a closed-loop optimization mechanism between the model and the user, achieving continuous learning and performance improvement.

[0195] Specifically, the step of obtaining feedback system incremental learning to continuously improve detection accuracy based on system prediction results and user interaction data includes (i.e., step 8 specifically includes):

[0196] Step 8.1: After the system outputs the anomaly determination result, a dialog box will pop up to ask the user and display the candidate anomaly semantic labels. The user can choose to confirm the system's prediction result or correct a more accurate anomaly label.

[0197] Specifically, in step 8.1, in a family living room environment, video surveillance captures a bright spot area where flames flicker, an audio sensor detects crackling burning sounds, and an infrared sensor detects a continuously rising temperature signal. The system initially determines this to be "abnormal lighting" and generates candidate abnormal semantic tags: intrusion (similarity 0.80), falling (similarity 0.68), sound of breaking windows (similarity 0.55), and loud cries for help (0.20). The interactive interface prompts the user: "Possible abnormal event detected. Please confirm whether it is an intrusion or select another tag."

[0198] Step 8.2: The user confirmation result is converted into a new anomaly label and stored in the feedback database along with the corresponding multimodal feature vector.

[0199] Specifically, in step 8.2, the user confirms that the actual situation is "fire". The system stores the confirmation result in the feedback database, including: tag: fire, video features: high brightness fluctuation in the flame area + smoke diffusion features, audio features: crackling sound of burning, low frequency energy continuously increasing, infrared features: temperature curve continuously rising, local infrared intensity abnormally concentrated.

[0200] Step 8.3: Based on the feedback labels, the system calls Step 1 to update the semantic library, adds new semantic representations, and calls Step 6 to incrementally learn and update the model according to the feedback data to adapt to the new feature-label correspondence.

[0201] Specifically, in step 8.3, the system calls the text encoder to generate a semantic vector for "fire" and adds it to the anomaly semantic library. The original semantic library is: {intrusion, fall, sound of breaking windows, loud cries for help}, and the updated semantic library is: {intrusion, fall, sound of breaking windows, loud cries for help, fire}. The system performs incremental learning based on feedback samples to update the weights of the anomaly classification model, enabling it to more accurately distinguish between "fire" and "event".

[0202] Step 8.4: Through multiple rounds of interaction, the system continuously collects user feedback, dynamically expands the semantic library and updates the model, forming a closed-loop adaptive mechanism for human-machine collaboration.

[0203] Specifically, in step 8.4, when a similar fire occurs again, the system can quickly identify the event as a "fire" and will not misjudge it as "abnormal lighting." Through multiple user feedbacks, the system continuously expands its anomaly type library and improves detection accuracy, achieving intelligent and adaptive anomaly recognition in home scenarios.

[0204] In another specific embodiment, such as Figure 3 As shown, in this embodiment of the invention, the abnormal event labels are used to generate semantic vectors via a Sentence-BERT encoder; the multimodal data stream is processed by a ResNet, VGGish, and infrared encoder to generate video, audio, and infrared features, respectively; the semantic vectors are concatenated with the features of each modality and then input into an MLP confidence network to obtain modality confidence vectors; a fusion strategy vector is generated by a reinforcement learning scheduler (DDPG) to perform weighted fusion of all modality features; the vectors are then input into a Transformer classifier to obtain anomaly detection results; and the semantic database, confidence model, and reward function are updated in conjunction with user feedback.

[0205] The multimodal home security anomaly detection method provided in this invention first acquires multimodal home monitoring data, including video data, audio data, and infrared data. Then, it extracts multiple modality mapping features from the multimodal home monitoring data and concatenates and fuses each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor. The candidate anomaly semantic vector is generated based on a pre-trained text model. Finally, the multimodal feature tensor is input into a trained lightweight neural network model, which analyzes the modalities and candidate anomalies within the multimodal feature tensor. The semantic matching degree is evaluated, and the confidence weight of each modality is output. The trained lightweight neural network model is trained based on the semantic alignment loss function. Then, the confidence weight of each modality is optimized through DDPG reinforcement learning to obtain the optimized confidence weight of each modality. The DDPG reinforcement learning uses detection accuracy, confidence variance, and feedback consistency as reward functions. Finally, based on the optimized confidence weights of all modalities and the mapping features of all modalities, the identification result of home security anomaly detection is obtained through a pre-trained Transformer sequence modeling network. This invention differs from methods that rely solely on single-modality information for anomaly detection by acquiring multimodal data from home surveillance. It introduces anomaly semantic vectors generated from a pre-trained text model as guiding information, which are then concatenated and fused with multimodal mapping features. This achieves deep coupling between anomaly semantics and multimodal perception features, effectively improving the ability to discriminate specific anomaly types. Through a semantic alignment loss function, a lightweight neural network model is trained to estimate the matching confidence of each modality with candidate anomaly semantics, achieving semantically guided modality selection optimization. This overcomes the adaptability bottleneck of traditional fixed or attention-based fusion methods. Furthermore, the DDPG reinforcement learning framework, with detection accuracy, confidence variance, and feedback consistency as reward functions, further optimizes the confidence weights of modalities, enabling adaptive scheduling strategies across scenarios and anomaly categories. This effectively improves robustness and flexibility in complex and changing environments, significantly increasing the accuracy of anomaly detection in complex environments. This solves the problem that existing home security anomaly detection methods often rely on single-modality detection, leading to low accuracy in complex environments.

[0206] Example 2:

[0207] like Figure 4 As shown, this embodiment provides a multimodal home security anomaly detection device for performing the above-described multimodal home security anomaly detection method, including:

[0208] The acquisition module 11 is used to acquire home monitoring multimodal data, wherein the home monitoring multimodal data includes: video data, audio data, and infrared data;

[0209] The extraction and splicing module 12 is connected to the acquisition module 11 and is used to extract multiple modality mapping features from the home monitoring multimodal data, and splice and fuse each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor, wherein the candidate anomaly semantic vector is generated based on a pre-trained text model;

[0210] The input evaluation module 13, connected to the extraction and splicing module 12, is used to input the multimodal feature tensor into the trained lightweight neural network model. The lightweight neural network model evaluates the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics, and outputs the confidence weight of each modality. The trained lightweight neural network model is trained based on the semantic alignment loss function.

[0211] The optimization module 14, connected to the input evaluation module 13, is used to optimize the confidence weight of each modality through DDPG reinforcement learning to obtain the optimized confidence weight of each modality, wherein the DDPG reinforcement learning uses detection accuracy, confidence variance and feedback consistency as reward functions;

[0212] Module 15, connected to the optimization module 14, is used to obtain the identification result of home security anomaly detection through a pre-trained Transformer sequence modeling network based on the confidence weights of all the optimized modalities and the mapping features of all the modalities.

[0213] Furthermore, the device also includes:

[0214] The first definition module is used to define multiple exception event labels;

[0215] The conversion module is invoked to convert each of the abnormal event labels into an abnormal semantic vector by invoking the pre-trained text model, thereby forming an abnormal semantic vector representation library.

[0216] Furthermore, the acquisition module 11 specifically includes:

[0217] The acquisition and synchronization unit is used to acquire the video data, audio data, and infrared data, and synchronize the time domain of the video data, audio data, and infrared data based on a unified timestamp mechanism.

[0218] Furthermore, the extraction and splicing module 12 specifically includes:

[0219] The first extraction unit is used to input video data into a pre-trained ResNet network and extract video feature vectors.

[0220] The second extraction unit is used to input audio data into the VGGish audio encoder and extract audio feature vectors.

[0221] The third extraction unit is used to input infrared data into a dedicated convolutional encoder and extract infrared feature vectors.

[0222] The unit is used to obtain each modality mapping feature by passing through modality-independent fully connected layers based on the video feature vector, audio feature vector, and infrared feature vector, wherein all the modality mapping features have the same dimension;

[0223] The concatenation and stacking unit is used to concatenate each modality mapping feature with the current candidate anomaly semantic vector in the anomaly semantic vector representation library, and stack each concatenated modality mapping feature to obtain the multimodal feature tensor.

[0224] Furthermore, the device also includes:

[0225] The acquisition and calculation module is used to acquire the confidence weights of multiple historical modalities output by the lightweight neural network model based on the historical multimodal feature tensor, and to calculate the cosine similarity between the mapping feature of each historical modality and the corresponding anomaly semantic vector.

[0226] The second definition module is used to define a semantic alignment loss function based on the cosine similarity between each historical modality mapping feature and the corresponding abnormal semantic vector.

[0227] The training module is used to train the lightweight neural network model based on the semantic alignment loss function and the classification cross-entropy loss function corresponding to the confidence weights of all the historical modalities, so as to obtain the trained lightweight neural network model.

[0228] Furthermore, the device also includes:

[0229] The design module is used to design detection accuracy, confidence variance, and feedback consistency as reward functions;

[0230] The learning module is used to learn the optimal strategy for optimizing the confidence weights of each modality based on the reward function using the DDPG reinforcement learning algorithm.

[0231] Furthermore, the device also includes:

[0232] The learning and recognition module is used to continuously learn and recognize anomaly types and device status based on the semantic closed-loop learning process built through user interaction feedback mechanism, the anomaly semantic vector representation library dynamically expanded through incremental learning, and the confidence weight of each modality after historical modeling and Bayesian updates using Beta distribution.

[0233] Example 3:

[0234] refer to Figure 5 This embodiment provides a multimodal home security anomaly detection device, including a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the multimodal home security anomaly detection method in Embodiment 1.

[0235] The memory 21 is connected to the processor 22. The memory 21 can be a flash memory, a read-only memory or other memory, and the processor 22 can be a central processing unit or a microcontroller.

[0236] Example 4:

[0237] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal home security anomaly detection method in Embodiment 1 above.

[0238] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules or other data. Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0239] In summary, the multimodal home security anomaly detection method, device, and medium provided in this embodiment of the invention first acquire multimodal home monitoring data, including video data, audio data, and infrared data; then, multiple modality mapping features are extracted from the multimodal home monitoring data, and each modality mapping feature is concatenated and fused with a candidate anomaly semantic vector to obtain a multimodal feature tensor, wherein the candidate anomaly semantic vector is generated based on a pre-trained text model; finally, the multimodal feature tensor is input into a trained lightweight neural network model, and the lightweight neural network model analyzes each modality in the multimodal feature tensor. The matching degree between the modality and the candidate anomaly semantics is evaluated, and the confidence weight of each modality is output. The trained lightweight neural network model is trained based on the semantic alignment loss function. Then, the confidence weight of each modality is optimized through DDPG reinforcement learning to obtain the optimized confidence weight of each modality. The DDPG reinforcement learning uses detection accuracy, confidence variance, and feedback consistency as reward functions. Finally, based on the optimized confidence weights of all modalities and the mapping features of all modalities, the identification result of home security anomaly detection is obtained through a pre-trained Transformer sequence modeling network. This invention differs from methods that rely solely on single-modality information for anomaly detection by acquiring multimodal data from home surveillance. It introduces anomaly semantic vectors generated from a pre-trained text model as guiding information, which are then concatenated and fused with multimodal mapping features. This achieves deep coupling between anomaly semantics and multimodal perception features, effectively improving the ability to discriminate specific anomaly types. Through a semantic alignment loss function, a lightweight neural network model is trained to estimate the matching confidence of each modality with candidate anomaly semantics, achieving semantically guided modality selection optimization. This overcomes the adaptability bottleneck of traditional fixed or attention-based fusion methods. Furthermore, the DDPG reinforcement learning framework, with detection accuracy, confidence variance, and feedback consistency as reward functions, further optimizes the confidence weights of modalities, enabling adaptive scheduling strategies across scenarios and anomaly categories. This effectively improves robustness and flexibility in complex and changing environments, significantly increasing the accuracy of anomaly detection in complex environments. This solves the problem that existing home security anomaly detection methods often rely on single-modality detection, leading to low accuracy in complex environments.

[0240] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A multimodal home security anomaly detection method, characterized in that, The method includes: Acquire multimodal data from home surveillance, wherein the multimodal data from home surveillance includes: video data, audio data, and infrared data; Multiple modality mapping features are extracted from the home surveillance multimodal data, and each modality mapping feature is concatenated and fused with a candidate anomaly semantic vector to obtain a multimodal feature tensor, wherein the candidate anomaly semantic vector is generated based on a pre-trained text model; The multimodal feature tensor is input into a trained lightweight neural network model. The lightweight neural network model evaluates the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics, and outputs the confidence weight of each modality. The trained lightweight neural network model is trained based on the semantic alignment loss function. The confidence weights of each modality are optimized by DDPG reinforcement learning to obtain the optimized confidence weights of each modality, wherein the DDPG reinforcement learning uses detection accuracy, confidence variance and feedback consistency as reward functions; Based on the optimized confidence weights of all the modalities and the mapping features of all the modalities, the identification results of home security anomaly detection are obtained through a pre-trained Transformer sequence modeling network; Before acquiring the home surveillance multimodal data, the method further includes: Define multiple exception event labels; By calling the pre-trained text model, each abnormal event label is converted into an abnormal semantic vector, thus forming an abnormal semantic vector representation library; The candidate anomaly semantic vector is at least one anomaly semantic vector from the anomaly semantic vector representation library predefined during system initialization in the home security monitoring system.

2. The method according to claim 1, characterized in that, The acquisition of multimodal data from home surveillance specifically includes: The video data, audio data, and infrared data are collected, and the time domain of the video data, audio data, and infrared data is synchronized based on a unified timestamp mechanism.

3. The method according to claim 1, characterized in that, The step of extracting multiple modality mapping features from the home surveillance multimodal data and concatenating and fusing each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor specifically includes: The video data is input into a pre-trained ResNet network to extract video feature vectors; The audio data is input into the VGGish audio encoder to extract the audio feature vector; Infrared data is input into a dedicated convolutional encoder to extract infrared feature vectors; Based on the video feature vector, audio feature vector, and infrared feature vector, each modality mapping feature is obtained through a modality-independent fully connected layer, wherein all the modality mapping features have the same dimension; Each modality mapping feature is concatenated with the current candidate anomaly semantic vector confirmed by the user from the candidate anomaly semantic vector representation library, and the concatenated modality mapping features are stacked to obtain the multimodal feature tensor.

4. The method according to claim 1, characterized in that, Before inputting the multimodal feature tensor into a trained lightweight neural network model, evaluating the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics through the lightweight neural network model, and outputting the confidence weight of each modality, the method further includes: Obtain the confidence weights of multiple historical modalities from the output of the historical multimodal feature tensor of the lightweight neural network model, and calculate the cosine similarity between the mapped features of each historical modality and the corresponding anomaly semantic vector. A semantic alignment loss function is defined based on the cosine similarity between each historical modality mapping feature and the corresponding abnormal semantic vector; The lightweight neural network model is trained based on the semantic alignment loss function and the classification cross-entropy loss function corresponding to the confidence weights of all the historical modalities, resulting in the trained lightweight neural network model.

5. The method according to claim 1, characterized in that, Before optimizing the confidence weights of each modality using DDPG reinforcement learning to obtain the optimized confidence weights of each modality, the method further includes: The detection accuracy, confidence variance, and feedback consistency are designed as reward functions. Based on the reward function, the DDPG reinforcement learning algorithm is used to learn the optimal strategy for optimizing the confidence weights of each modality.

6. The method according to claim 1, characterized in that, After obtaining the identification result of home security anomaly detection through a pre-trained Transformer sequence modeling network based on the optimized confidence weights of all the modalities and all the modal mapping features, the method further includes: Based on the semantic closed-loop learning process built through user interaction feedback mechanism, the abnormal semantic vector representation library dynamically expanded through incremental learning, and the confidence weight of each modality after introducing Beta distribution for historical modeling and Bayesian update, continuous learning and recognition of abnormal types and device status are achieved.

7. A multimodal home security anomaly detection device, characterized in that, include: The acquisition module is used to acquire multimodal data from home surveillance, wherein the multimodal data from home surveillance includes: video data, audio data, and infrared data; An extraction and splicing module, connected to the acquisition module, is used to extract multiple modality mapping features from the home monitoring multimodal data, and splice and fuse each modality mapping feature with a candidate anomaly semantic vector to obtain a multimodal feature tensor, wherein the candidate anomaly semantic vector is generated based on a pre-trained text model; An input evaluation module, connected to the extraction and splicing module, is used to input the multimodal feature tensor into a trained lightweight neural network model. The lightweight neural network model evaluates the matching degree between each modality in the multimodal feature tensor and the candidate anomaly semantics, and outputs the confidence weight of each modality. The trained lightweight neural network model is trained based on a semantic alignment loss function. An optimization module, connected to the input evaluation module, is used to optimize the confidence weight of each modality through DDPG reinforcement learning to obtain the optimized confidence weight of each modality, wherein the DDPG reinforcement learning uses detection accuracy, confidence variance and feedback consistency as reward functions; The module is connected to the optimization module and is used to obtain the identification result of home security anomaly detection through a pre-trained Transformer sequence modeling network based on the confidence weights of all the optimized modalities and the mapping features of all the modalities. Furthermore, the device also includes: The first definition module is used to define multiple exception event labels; The conversion module is invoked to convert each of the abnormal event labels into abnormal semantic vectors by invoking the pre-trained text model, thereby forming an abnormal semantic vector representation library. The candidate anomaly semantic vector is at least one anomaly semantic vector from the anomaly semantic vector representation library predefined during system initialization in the home security monitoring system.

8. A multimodal home security anomaly detection device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the multimodal home security anomaly detection method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multimodal home security anomaly detection method as described in any one of claims 1-6.

Images