Abnormal home state signal detection method and system based on multimodal sensing

Abstract

The present invention provides a method and system for detecting abnormal state signals at home based on multimodal sensing, which relates to the field of detection technology. The method comprises collecting human motion, acoustic and environmental parameters through millimeter wave radar, acoustic sensors and environmental sensors, inputting the parameters into a deep fusion network, extracting spatiotemporal feature sequences using a spatiotemporal attention mechanism and a multi-scale convolutional neural network, and forming joint probability features in combination with the results of acoustic feature spectrum analysis. The joint probability features are input into a time series knowledge graph, and after being processed by a graph attention network and a dual-stream autoencoder network, scene-adaptive features are obtained. Finally, a multi-task learning network is combined with a risk propagation neural network and a recursive neural network to perform anomaly prediction and risk level assessment, and output an early warning strategy in combination with historical early warning information. The present invention can effectively fuse multimodal data, improve the accuracy and reliability of abnormal state detection at home, and provide more accurate risk assessment and early warning strategies.

Description

Technical Field

[0001] The present invention relates to detection technology, and in particular to a method and system for detecting abnormal state signals at home based on multimodal sensing. Background Art

[0002] Home abnormal state detection technology aims to use sensors to detect abnormal events in the home environment, such as falls, fires, and intrusions, and issue timely alarms to ensure the safety of groups such as the elderly and children, as well as the safety of family property. Existing home abnormal state detection technologies mainly rely on single sensors or simple multi-sensor fusion methods, such as relying solely on camera image recognition or sound recognition. These methods have the following shortcomings:

[0003] Poor robustness: A single sensor is susceptible to environmental interference, such as lighting changes and noise, which can lead to reduced detection accuracy. Simple multi-sensor fusion methods also fail to fully utilize the complementary information between different sensors, resulting in poor detection results in complex scenarios.

[0004] Lack of scene adaptation capabilities: Existing methods are usually based on pre-set rules or models, which are difficult to adapt to the differences in different home environments, such as furniture layout, people's activity habits, etc., resulting in a high false alarm rate.

[0005] Inadequate risk assessment capabilities: Existing methods typically focus solely on detecting abnormal events, but lack the ability to assess risk levels. For example, the same fall poses different risks to elderly and young people. This lack of risk assessment capabilities can lead to ineffective interventions. Summary of the Invention

[0006] The embodiments of the present invention provide a method and system for detecting abnormal home status signals based on multimodal sensing, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a method for detecting abnormal home status signals based on multimodal sensing, including:

[0009] The collected human motion data, acoustic features, and environmental parameters are input into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatiotemporal attention mechanism, extracts spatiotemporal feature sequences through a multi-scale convolutional neural network, and combines them with the spectral analysis results of the acoustic features to form a joint probability feature.

[0010] The joint probability features are processed by the temporal knowledge graph of the graph attention network to obtain multidimensional combined features, and the multidimensional combined features are input into the two-stream autoencoder network. Feature mapping is performed on the source scene and adversarial training is used to obtain target scene features. The source scene features and target scene features are input into the memory-enhanced neural network, and the scene adaptive features are output in combination with the reinforcement learning method.

[0011] The scenario-adaptive features are input into a multi-task learning network, comprehensive features are extracted through the shared representation layer, anomaly prediction results are output through the task attention layer, anomaly prediction results are input into a risk propagation neural network, the coupling relationship of risk factors is analyzed through a message passing mechanism, and the risk level is output; the risk level is input into a recursive neural network, and feature memory is performed through a gating mechanism in combination with historical warning feedback information, and a warning strategy is output.

[0012] In an optional embodiment,

[0013] The steps of inputting the collected human motion data, acoustic features, and environmental parameters into the deep fusion network to form joint probability features include:

[0014] Human motion data, acoustic signals, and environmental parameters are filtered, transformed, and normalized to obtain standardized features. The standardized features are sequentially passed through the temporal attention layer and modal attention layer of a two-layer attention network to obtain weighted multimodal features. The multimodal features are input into a multi-branch convolutional neural network, and a spatiotemporal feature sequence is extracted through depthwise separable convolution and residual connection.

[0015] The spatiotemporal feature sequence is input into a Gaussian mixture model, which includes multiple Gaussian components, each Gaussian component has a corresponding mixing weight, a mean vector and a covariance matrix, and the mean vector and the covariance matrix are both functions of latent variables. The latent variables are recursively updated through a long short-term memory network to obtain a spatiotemporal feature probability distribution; the spatiotemporal feature probability distribution and the acoustic features are input into a conditional variational autoencoder, which includes a two-branch encoder network. The conditional variational autoencoder obtains a high-level feature representation through the two-branch encoder network, and outputs posterior distribution parameters through a mean network and a variance network; the posterior distribution parameters are reparameterized and sampled and reconstructed through a decoder network, and the evidence lower bound of the reconstruction term and the KL divergence is calculated; the gradient norm of the evidence lower bound on the spatiotemporal feature probability distribution and the acoustic feature is calculated to obtain a modal importance score, the KL divergence weight is dynamically adjusted and the evidence lower bound is maximized, and the conditional variational autoencoder is trained to obtain a joint probability feature of action and sound.

[0016] In an optional embodiment,

[0017] The steps of processing the joint probability features through the temporal knowledge graph of the graph attention network to obtain multidimensional combined features, inputting the multidimensional combined features into the two-stream autoencoder network, performing feature mapping on the source scene, and obtaining target scene features through adversarial training include:

[0018] The joint probability features are input into the time series knowledge graph, and the correlation weights between time series nodes are calculated through the attention mechanism. The time series nodes correspond to feature vectors at different moments. The weighted time series knowledge graph is processed by the multi-head attention mechanism of the multi-layer graph attention network, and multi-dimensional combined features are obtained by multi-layer stacking.

[0019] Inputting the multidimensional combined features into a two-stream autoencoder network, wherein the source scene encoder branch is composed of a multi-layer perceptron, encoding the input features into source scene latent features through layer-by-layer nonlinear transformation, and the target scene adapter branch introduces a residual connection in the encoder, adding the source scene latent features and the encoder output features of the target scene adapter branch to obtain the target scene latent features, and reconstructing the target scene features through the decoder network;

[0020] Inputting the target scene features into a discriminant network with a block discriminant structure, performing local discrimination on the features through multi-layer convolution, calculating the Wasserstein distance between the source scene potential features and the target scene potential features as the adversarial loss and introducing a gradient penalty term;

[0021] The maximum average difference between the latent features of the source scene and the target scene is calculated as a distribution alignment constraint, and reconstruction constraints, cycle consistency constraints and contrastive learning losses are introduced at the same time;

[0022] The adversarial loss of the discriminator network and the joint loss of the target scene adapter are optimized alternately by exponential sliding average. The joint loss includes adversarial loss, distribution alignment loss, cycle consistency loss and contrastive loss.

[0023] In an optional embodiment,

[0024] The steps of inputting source scene features and target scene features into a memory-enhanced neural network and combining the reinforcement learning method to output scene adaptive features include:

[0025] The source scene features and target scene features are input into the hierarchical memory writing mechanism of the memory-augmented neural network for feature normalization. The fused features are obtained through interaction through the multi-head attention mechanism. The target memory slot is determined based on local sensitive hashing, and the fused features are written into the memory module through the gated writing mechanism.

[0026] The memory module is input into a reinforcement learning framework, which includes an actor-critic network structure and a multi-step memory reasoning mechanism. The actor network generates a memory operation strategy based on the current memory content, the fusion features, and the historical operation sequence. The critic network uses a dual value evaluation network structure to evaluate the state action value function, constructs a reward function based on feature mutual information, distribution distance, and memory timeliness, adopts a policy gradient update strategy, and performs multi-step memory reasoning through temporal difference.

[0027] Constructing a multi-scale query vector for the fused features, the multi-scale query vector including a spatial scale vector, a temporal scale vector, and a channel scale vector, retrieving a memory feature having a maximum cosine similarity with the query vector from the memory module, integrating the feature with the fused feature through an adaptive fusion network, and obtaining a scene-adaptive feature through a residual connection;

[0028] The memory module is optimized based on a weighted combination of memory duration and feature mutual information, including priority update, memory compression and dynamic adjustment of memory capacity.

[0029] In an optional embodiment,

[0030] The memory module is input into a reinforcement learning framework, which includes an Actor-Critic network structure and a multi-step memory reasoning mechanism. The Actor network generates a memory operation strategy based on the current memory content, the fusion features, and the historical operation sequence. The Critic network uses a dual value evaluation network structure to evaluate the state action value function. A reward function is constructed based on feature mutual information, distribution distance, and memory timeliness. The policy gradient update strategy is used and multi-step memory reasoning is performed through temporal difference. The steps include:

[0031] The Actor network encodes the current memory content through a multi-layer perceptron to obtain a memory representation, performs feature transformation on the fused features through a linear transformation to obtain a fused feature representation, and uses a gated recurrent unit network to encode the historical operation sequence to obtain a history representation. The memory representation, fused feature representation, and history representation are concatenated and passed through a fully connected layer to obtain the memory operation strategy distribution. A dual value evaluation network is used to construct the Critic network, including a feature value network based on feature mutual information and distribution distance, and a memory value network based on storage duration and usage frequency. The state-action value function is obtained by combining the outputs of the feature value network and the memory value network through a learnable balance factor.

[0032] Construct a comprehensive reward function based on multi-factor coupling, including a mutual information measurement term between features and memory, a distribution distance term between source and target scene features, and a memory timeliness measurement term. The weight coefficients are determined through grid search and cross-validation, and the weights are dynamically adjusted based on the mutual information change rate, distribution distance change rate, and memory timeliness change rate.

[0033] Cumulative rewards are constructed using n-step temporal differences, and the Actor network is updated based on the rewards from the current to the next n states and the decay value of the state-action-value function. The memory operation strategy is updated iteratively through temporal difference errors. A memory tracking mechanism based on importance sampling is introduced to calculate the importance weight of each state in the historical state sequence. The importance weight is the ratio of the current strategy probability to the behavior strategy probability. The cumulative product of the importance weight and the decay factor is used as the state importance measure, and memory operations are tracked based on the state importance.

[0034] In an optional embodiment,

[0035] The steps of inputting the scenario adaptive features into a multi-task learning network, extracting comprehensive features through a shared representation layer, outputting anomaly prediction results through a task attention layer, inputting the anomaly prediction results into a risk propagation neural network, analyzing the coupling relationship of risk factors through a message passing mechanism, and outputting a risk level include:

[0036] Based on scene-adaptive features, local-to-global features are extracted through the multi-scale convolution kernels of the shared representation layer of the multi-task learning network, and then integrated into a feature pyramid network to obtain comprehensive features. A multi-head collaborative attention mechanism is used to calculate the attention score, and the multi-head outputs are integrated and fused with the comprehensive features through a residual gating mechanism to obtain anomaly prediction results. The anomaly prediction results are input into the risk propagation neural network to construct a risk factor relationship graph. The risk factor nodes and the correlation between factors are processed through the node feature network and the edge feature network respectively, to obtain the edge features of the initial node features and the correlation between risk factors.

[0037] A message passing mechanism based on gated graph convolution is adopted. The message generation submodule combines source node features, target node features, and edge features through a gating unit, and the message aggregation submodule performs weighted aggregation of neighborhood message features based on the attention coefficient between nodes.

[0038] Multiple rounds of message passing are performed on the risk factor relationship graph to iteratively update the node features, and the risk factor node features after a preset number of rounds of message passing are added to the initial features of the risk factor nodes after residual mapping to obtain the risk factor node state; the risk propagation intensity between the risk factor nodes is calculated based on the feature similarity between the risk factor node states and the corresponding edge features, and the risk level is obtained through hierarchical graph pooling integration.

[0039] In an optional embodiment,

[0040] The steps of inputting the risk level into a recursive neural network, combining historical warning feedback information with feature memory through a gating mechanism, and outputting a warning strategy include:

[0041] The risk level is encoded through a bidirectional long short-term memory network to obtain a fused temporal representation. The historical warning feedback information is input into the feature transformation network to obtain the feedback feature representation. Based on the fused temporal representation and feedback feature representation, the attention weights are calculated at each layer of the time granularity pyramid, and the multi-granularity memory features are obtained through weighted integration through the soft attention mechanism.

[0042] A forget gate and an update gate are designed to dynamically update the multi-granularity memory features, wherein the forget gate controls the degree of retention of historical information based on the fused temporal representation and the multi-granularity memory features, and the update gate controls the proportion of new knowledge introduced, thereby obtaining an updated memory feature. The fused temporal representation and the updated memory feature are input into a gated recurrent unit for adaptive fusion to obtain a warning state representation.

[0043] A multi-objective early warning optimization function is constructed, which comprehensively considers the timeliness of early warning, early warning accuracy and abnormal behavior recognition rate. The early warning state representation is input into the deep Q learning network, and a state-action value function is constructed based on the scene adaptive features and early warning response behavior. The state of the state-action value function includes the scene adaptive features and environmental parameters, and the action of the state-action value function includes the adjustment of the early warning level and the recommendation of intervention measures. The state-action value function is mapped to obtain the early warning strategy, and the early warning strategy is iteratively optimized by the temporal difference algorithm to obtain the optimal early warning strategy.

[0044] According to a second aspect of the embodiments of the present invention,

[0045] Provides a home abnormal state signal detection system based on multimodal sensing, including:

[0046] The first unit is used to input the collected human motion data, acoustic features and environmental parameters into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatiotemporal attention mechanism, extracts spatiotemporal feature sequences through a multi-scale convolutional neural network, and combines them with the spectral analysis results of the acoustic features to form a joint probability feature;

[0047] The second unit is used to process the joint probability features through the temporal knowledge graph of the graph attention network to obtain multidimensional combined features, input the multidimensional combined features into the two-stream autoencoder network, perform feature mapping on the source scene and use adversarial training to obtain target scene features; input the source scene features and target scene features into the memory-enhanced neural network, and output scene-adaptive features in combination with the reinforcement learning method;

[0048] The third unit is used to input the scenario adaptive features into the multi-task learning network, extract comprehensive features through the shared representation layer, output the anomaly prediction results through the task attention layer, input the anomaly prediction results into the risk propagation neural network, analyze the coupling relationship of risk factors through the message passing mechanism and output the risk level; input the risk level into the recursive neural network, combine the historical warning feedback information with the feature memory through the gating mechanism and output the warning strategy.

[0049] According to a third aspect of the embodiments of the present invention,

[0050] An electronic device is provided, comprising:

[0051] processor;

[0052] a memory for storing processor-executable instructions;

[0053] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0054] According to a fourth aspect of the embodiments of the present invention,

[0055] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0056] The present invention integrates human motion, acoustic characteristics and environmental parameters through multimodal sensing, and uses a deep learning network for feature extraction and analysis, which can more comprehensively capture signals of abnormal states, thereby improving detection accuracy and reducing false alarm and missed alarm rates.

[0057] The present invention adopts a dual-stream autoencoder network and a memory-enhanced neural network, which can be adaptively adjusted according to different scenarios, and combines risk propagation neural networks and recursive neural networks to perform risk level assessment and early warning strategy generation, achieving more accurate risk warnings.

[0058] This paper utilizes temporal knowledge graphs and multi-task learning networks to better capture the spatiotemporal correlations and potential risk factors between data, and enhances the interpretability of the system through graph attention networks and message passing mechanisms, thereby improving the robustness and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of a flow chart of a method for detecting abnormal home state signals based on multimodal sensing according to an embodiment of the present invention;

[0060] Figure 2 This is a comparison chart of adaptability after scene conversion;

[0061] Figure 3Monte Carlo simulation results. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0063] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0064] Figure 1 FIG. 1 is a flow chart of a method for detecting abnormal home state signals based on multimodal sensing according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0065] Human motion data is collected through millimeter-wave radar, acoustic features are collected through acoustic sensors, and environmental parameters are collected through temperature, humidity, and air quality sensors. The human motion data, acoustic features, and environmental parameters are input into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatiotemporal attention mechanism. The weight assignment results are input into a multi-scale convolutional neural network to extract a spatiotemporal feature sequence. The spatiotemporal feature sequence is combined with the spectral analysis results of the acoustic features to form a joint probability feature.

[0066] The joint probability features are input into a temporal knowledge graph, which performs feature interaction operations through a graph attention network and outputs multidimensional combined features. The multidimensional combined features are input into a two-stream autoencoder network, the first branch of which performs feature mapping on the source scene, and the second branch calculates the target scene features through adversarial training; the source scene features and the target scene features are input into a memory-enhanced neural network, which outputs scene-adaptive features in combination with a reinforcement learning method;

[0067] The scenario-adaptive features are input into a multi-task learning network, which extracts comprehensive features through a shared representation layer and outputs anomaly prediction results through a task attention layer. The anomaly prediction results are input into a risk propagation neural network for risk propagation analysis, and the coupling relationship of risk factors is analyzed through a message passing mechanism to output a risk level. The risk level is input into a recursive neural network, which combines historical warning feedback information with a gating mechanism to perform feature memory and output a warning strategy.

[0068] In an optional embodiment,

[0069] The steps of inputting the collected human motion data, acoustic features, and environmental parameters into the deep fusion network to form joint probability features include:

[0070] Human motion data, acoustic signals, and environmental parameters are filtered, transformed, and normalized to obtain standardized features. The standardized features are sequentially passed through the temporal attention layer and modal attention layer of a two-layer attention network to obtain weighted multimodal features. The multimodal features are input into a multi-branch convolutional neural network, and a spatiotemporal feature sequence is extracted through depthwise separable convolution and residual connection.

[0071] The spatiotemporal feature sequence is input into a Gaussian mixture model, which includes multiple Gaussian components, each Gaussian component has a corresponding mixing weight, a mean vector and a covariance matrix, and the mean vector and the covariance matrix are both functions of latent variables. The latent variables are recursively updated through a long short-term memory network to obtain a spatiotemporal feature probability distribution; the spatiotemporal feature probability distribution and the acoustic features are input into a conditional variational autoencoder, which includes a two-branch encoder network. The conditional variational autoencoder obtains a high-level feature representation through the two-branch encoder network, and outputs posterior distribution parameters through a mean network and a variance network; the posterior distribution parameters are reparameterized and sampled and reconstructed through a decoder network, and the evidence lower bound of the reconstruction term and the KL divergence is calculated; the gradient norm of the evidence lower bound on the spatiotemporal feature probability distribution and the acoustic feature is calculated to obtain a modal importance score, the KL divergence weight is dynamically adjusted and the evidence lower bound is maximized, and the conditional variational autoencoder is trained to obtain a joint probability feature of action and sound.

[0072] Exemplarily, the collected multimodal data is preprocessed. Human motion data can come from a depth sensor or an inertial sensor, which is converted into a three-dimensional joint coordinate sequence. Acoustic signals are collected using a microphone, and environmental parameters can be obtained using a temperature sensor, a humidity sensor, etc. These data of different modes are filtered, such as using a Kalman filter to remove noise, and then data transformation is performed, such as performing a short-time Fourier transform on the acoustic signal to obtain a spectrum diagram, and finally the data of all modes are normalized to the same numerical range, such as between 0 and 1, to form standardized features. For example, human motion data can be represented as the three-dimensional coordinates of each joint at each time point, the acoustic signal can be represented as the spectral energy of each time frame, and the environmental parameters can be represented as temperature and humidity values.

[0073] A two-layer attention network is used to dynamically assign weights to normalized features. The two-layer attention network consists of a temporal attention layer and a modal attention layer. The temporal attention layer learns weights for different time steps, for example, assigning higher weights to the beginning and end of an action. The modal attention layer learns weights for different modalities, for example, assigning higher weights to acoustic features when acoustic information is more important. Temporal and modal attention mechanisms can be implemented in various ways, such as using an attention mechanism based on scaled dot products. Assuming there are three modalities: motion, sound, and environment, the temporal attention layer can output a weight vector with a length of the number of time steps, and the modal attention layer can output a weight vector with a length of 3. Multiplying these two weight vectors yields the weight for each modality at each time step. These weights are applied to the normalized features to produce weighted multimodal features.

[0074] The weighted multimodal features are input into a multi-branch convolutional neural network to extract a spatiotemporal feature sequence. The multi-branch convolutional neural network uses convolution kernels of different sizes to capture spatiotemporal features of different scales. For example, small-sized convolution kernels are used to capture local detail features, while large-sized convolution kernels are used to capture global context information. Depthwise separable convolution can reduce the amount of computation, and residual connections can avoid the gradient vanishing problem, thereby improving the training efficiency and performance of the network. The output of the multi-branch convolutional neural network is a spatiotemporal feature sequence that contains feature information of different scales and modalities. Assuming the input feature dimension is 128 and the time step is 64, the output spatiotemporal feature sequence dimension can be 256 and the time step is 32.

[0075] The spatiotemporal feature sequence is input into a Gaussian mixture model (GMM) for probabilistic modeling. The GMM consists of multiple Gaussian components, each with its own weight, mean vector, and covariance matrix. These parameters are recursively updated using a long short-term memory (LSTM) network, which can capture long-term dependencies in time series data. Through LSTM updates, the GMM learns the probability distribution of spatiotemporal features. For example, if the GMM has 16 Gaussian components, each component has a weight, a 256-dimensional mean vector, and a 256x256 covariance matrix. The LSTM updates these parameters based on the current spatiotemporal feature sequence, thereby obtaining the probability distribution of the spatiotemporal features.

[0076] The probability distribution of spatiotemporal features and acoustic features are input into a conditional variational autoencoder (CVAE). The CVAE consists of an encoder and a decoder. The encoder encodes the spatiotemporal probability distribution and acoustic features into high-level feature representations and outputs the parameters of the posterior distribution. The decoder reconstructs the probability distribution of spatiotemporal features based on the sampled latent variables and acoustic features. The CVAE is trained by maximizing the evidence lower bound (ELBO). The ELBO consists of a reconstruction term and a KL divergence term. The reconstruction term measures the similarity of the reconstructed spatiotemporal probability distribution to the original distribution, while the KL divergence term measures the difference between the posterior distribution and the prior distribution. The modal importance score is obtained by calculating the gradient norm of the ELBO with respect to the spatiotemporal probability distribution and acoustic features. Dynamically adjusting the weight of the KL divergence term based on the modal importance score allows the model to focus more on important modal information. The output of the trained CVAE serves as the joint probability feature of motion and sound.

[0077] Through multimodal deep fusion and attention mechanism, the present invention can effectively fuse information from different modalities and improve the accuracy of action recognition, especially in complex scenarios. Through dynamic weight allocation and probabilistic modeling, it can effectively handle noise and missing data and enhance the robustness of the model. Through attention mechanism and probabilistic modeling, it can better understand the contribution of different modalities to action recognition and improve the interpretability of the model.

[0078] In an optional embodiment,

[0079] The steps of processing the joint probability features through the temporal knowledge graph of the graph attention network to obtain multidimensional combined features, inputting the multidimensional combined features into the two-stream autoencoder network, performing feature mapping on the source scene, and obtaining target scene features through adversarial training include:

[0080] The joint probability features are input into the time series knowledge graph, and the correlation weights between time series nodes are calculated through the attention mechanism. The time series nodes correspond to feature vectors at different moments. The weighted time series knowledge graph is processed by the multi-head attention mechanism of the multi-layer graph attention network, and multi-dimensional combined features are obtained by multi-layer stacking.

[0081] Inputting the multidimensional combined features into a two-stream autoencoder network, wherein the source scene encoder branch is composed of a multi-layer perceptron, encoding the input features into source scene latent features through layer-by-layer nonlinear transformation, and the target scene adapter branch introduces a residual connection in the encoder, adding the source scene latent features and the encoder output features of the target scene adapter branch to obtain the target scene latent features, and reconstructing the target scene features through the decoder network;

[0082] Inputting the target scene features into a discriminant network with a block discriminant structure, performing local discrimination on the features through multi-layer convolution, calculating the Wasserstein distance between the source scene potential features and the target scene potential features as the adversarial loss and introducing a gradient penalty term;

[0083] The maximum average difference between the latent features of the source scene and the target scene is calculated as a distribution alignment constraint, and reconstruction constraints, cycle consistency constraints and contrastive learning losses are introduced at the same time;

[0084] The adversarial loss of the discriminator network and the joint loss of the target scene adapter are optimized alternately by exponential sliding average. The joint loss includes adversarial loss, distribution alignment loss, cycle consistency loss and contrastive loss.

[0085] For example, a temporal knowledge graph is constructed. The input joint probability features are arranged in chronological order, and the feature vector at each moment constitutes a node in the graph. Then, the temporal correlation between the nodes is calculated. For example, the cosine similarity of the feature vectors of two nodes can be used to measure the correlation between them, and the calculation result is used as the weight of the edge connecting the two nodes. In this way, a time series knowledge graph with weights is constructed. Assuming that the dimension of the joint probability feature is 128 and the time step is 10, there are 10 nodes, and the feature dimension of each node is 128. The weights of the connection edges between nodes are, for example, 0.8, 0.5, 0.9, and so on.

[0086] The constructed temporal knowledge graph is fed into a multi-layer graph attention network. This network utilizes a multi-head attention mechanism to transform and aggregate node features. Each attention head focuses on different aspects of a node, and the results of multiple attention heads are combined to obtain a more comprehensive node representation. By stacking multiple layers, features can be transferred across multiple layers, ultimately generating multi-dimensional composite features. For example, after a three-layer graph attention network, the final output feature dimension is 512.

[0087] The resulting 512-dimensional multidimensional combined features are input into a two-stream autoencoder network. This network consists of a source scene encoder branch and a target scene adapter branch. The source scene encoder branch, composed of a multi-layer perceptron, encodes the input multidimensional combined features into low-dimensional source scene latent features, such as 256 dimensions, through layer-by-layer nonlinear transformations. The target scene adapter branch also contains an encoder with a similar structure to the source scene encoder, but with the introduction of a residual connection. The source scene latent features are added to the encoder output of the target scene adapter branch to obtain the target scene latent features. Finally, the target scene latent features are reconstructed through a decoder network to obtain the target scene features, with the same dimensionality as the input multidimensional combined features, i.e., 512 dimensions.

[0088] In order to train the target scene adapter, an adversarial training mechanism is introduced. The reconstructed target scene features are input into the discriminator network, which adopts a block discriminant structure and performs local discrimination on the features through multi-layer convolution. The Wasserstein distance between the source scene latent features and the target scene latent features is calculated as the adversarial loss. In order to stabilize the training process, a gradient penalty term is introduced in the adversarial training. At the same time, the maximum average difference measure between the source scene latent features and the target scene latent features is calculated as a distribution alignment constraint. For example, the feature distance is calculated using a Gaussian kernel function. In addition, a reconstruction constraint is imposed on the source scene latent features, and a cycle consistency constraint is imposed on the target scene latent features, that is, they are input into the target scene adapter again, and the output is required to be as close to the original input as possible. In order to enhance the discriminability of the features, a contrastive learning loss is also introduced.

[0089] The network is trained by jointly optimizing a loss function. Specifically, the reconstruction loss and contrastive loss of the source scene encoder branch are optimized, while the adversarial loss of the discriminator network and the joint loss of the target scene adapter are alternately optimized. The joint loss of the target scene adapter consists of the adversarial loss, the distribution alignment loss, the cycle consistency loss, and the contrastive loss. To stabilize the training process, an exponential moving average is used to update the parameters of the target scene adapter branch and the discriminator network.

[0090] The present invention effectively captures the temporal dependency of features through the temporal knowledge graph, and uses the dual-stream autoencoder and adversarial training mechanism to achieve effective adaptation of source scene features to target scene features, thereby improving the feature transfer learning ability across scenes and domains. Through the multi-layer graph attention network and multi-head attention mechanism, it can extract more comprehensive and robust feature representations, effectively reduce the influence of noise and irrelevant information, and improve the stability and generalization ability of feature representation. Through various mechanisms such as adversarial training, distribution alignment constraints, cycle consistency constraints and contrastive learning, the distribution of source scene features and target scene features is made as close as possible, thereby better achieving cross-scene feature adaptation and improving the transfer learning performance of the model.

[0091] In an optional embodiment,

[0092] The steps of inputting source scene features and target scene features into a memory-enhanced neural network and combining the reinforcement learning method to output scene adaptive features include:

[0093] The source scene features and target scene features are input into the hierarchical memory writing mechanism of the memory-augmented neural network for feature normalization. The fused features are obtained through interaction through the multi-head attention mechanism. The target memory slot is determined based on local sensitive hashing, and the fused features are written into the memory module through the gated writing mechanism.

[0094] The memory module is input into a reinforcement learning framework, which includes an actor-critic network structure and a multi-step memory reasoning mechanism. The actor network generates a memory operation strategy based on the current memory content, the fusion features, and the historical operation sequence. The critic network uses a dual value evaluation network structure to evaluate the state action value function, constructs a reward function based on feature mutual information, distribution distance, and memory timeliness, adopts a policy gradient update strategy, and performs multi-step memory reasoning through temporal difference.

[0095] Constructing a multi-scale query vector for the fused features, the multi-scale query vector including a spatial scale vector, a temporal scale vector, and a channel scale vector, retrieving a memory feature having a maximum cosine similarity with the query vector from the memory module, integrating the feature with the fused feature through an adaptive fusion network, and obtaining a scene-adaptive feature through a residual connection;

[0096] The memory module is optimized based on a weighted combination of memory duration and feature mutual information, including priority update, memory compression and dynamic adjustment of memory capacity.

[0097] Exemplarily, feature preprocessing and memory writing are performed. The source and target scene features are separately input into a memory-augmented neural network. To eliminate the influence of different feature dimensions and scales, the source and target scene features are normalized, for example, by scaling each feature dimension to zero mean and unit variance. A multi-head attention mechanism is then used to interact with the normalized source and target scene features to generate fused features. The multi-head attention mechanism allows the network to focus on feature interactions across different subspaces, thereby capturing richer feature representations. For example, if the source scene features are image feature vectors with a dimension of 1024, and the target scene features are text feature vectors with a dimension of 512, the multi-head attention mechanism can fuse them into a feature vector with a dimension of 2048, which incorporates the interaction information between the image and text. Next, a locality-sensitive hashing method is used to calculate multiple hash values ​​for the fused features, for example, eight hash values. Based on these hash values, the target memory slots are determined, for example, the fused features are written into the eight memory slots corresponding to the hash values. Finally, a gated write mechanism is used to write the fused features into the memory module. The gated write mechanism controls which information is written into the memory to prevent the memory from being overwritten by irrelevant information.

[0098] Reinforcement learning training is performed. The memory module is input into a reinforcement learning framework consisting of an actor-critic network structure and a multi-step memory inference mechanism. The actor network generates a memory operation policy based on the current memory content, fused features, and historical operation sequences, for example, deciding which information to read from or write to memory. The critic network uses a dual value evaluation network structure to evaluate the state-action value function: a feature value network assesses the value of the current state, and a memory value network assesses the value of the memory content. For example, the feature value network judges the scene's favorability based on the current scene's features, while the memory value network judges the potential benefit of the current action based on historical experience in memory. To guide reinforcement learning training, a comprehensive reward function is constructed based on the mutual information between features and memory, the distribution distance between source and target scene features, and a measure of memory timeliness. For example, a high reward is given if the information in memory is highly correlated with the current scene's features; a high reward is given if the source and target scene feature distributions differ significantly, encouraging the network to learn cross-scenario adaptability; and a high reward is given if the information in memory is recently written. Policy gradient methods are used to update memory operation policies. For example, the REINFORCE algorithm is used to update the parameters of the actor network. Multi-step memory reasoning is performed using temporal difference methods. For example, the TD(λ) algorithm is used to estimate long-term rewards. Memory tracking is performed based on the product of the accumulated importance of historical states and a decay factor. For example, different weights are assigned to past states, focusing on states that have a significant impact on current decisions.

[0099] Perform feature retrieval and fusion. Construct a multi-scale query vector for the fused features, including a spatial scale vector, a temporal scale vector, and a channel scale vector. For example, for image features, the spatial scale vector can represent different areas of the image, the temporal scale vector can represent different time frames of the image, and the channel scale vector can represent different channels of the image. Retrieve the memory feature with the maximum cosine similarity to the query vector from the memory module based on the multi-scale query vector. For example, calculate the cosine similarity between each query vector and all feature vectors in the memory, and select the feature vector with the highest similarity as the retrieval result. The retrieved memory features are integrated with the fused features through an adaptive fusion network, and a fully connected network is used to concatenate the fused features and the retrieved memory features and perform a nonlinear transformation. Finally, the scene adaptive features are obtained through residual connections, and the fused features and the output of the adaptive fusion network are added to obtain the final scene adaptive features.

[0100] Memory optimization. Memory modules are optimized based on a weighted combination of memory duration and feature mutual information, including priority updates, memory compression, and dynamic memory capacity adjustment. For example, memories with long duration and high mutual information with the current feature are prioritized; memories with short duration and low mutual information with the current feature are prioritized and compressed or deleted to free up memory space. Dynamic memory capacity adjustment automatically adjusts the size of memory modules based on task requirements and computing resource constraints.

[0101] By combining the information of the source scene and the target scene and utilizing the memory mechanism for knowledge accumulation and transfer, the present invention can extract more comprehensive and discriminative scene-adaptive features, effectively improving the feature representation capability. Through reinforcement learning training and memory optimization, it can adapt to changes in different scenes, improve the generalization ability of the model, and enable it to maintain good performance in new scenes. Through technologies such as local sensitive hashing and multi-scale query, it can quickly retrieve and access relevant information in memory, thereby improving computing efficiency and reducing computing costs.

[0102] In an optional embodiment,

[0103] The Actor network generates a memory operation strategy based on the current memory content, the fusion features, and the historical operation sequence. The Critic network uses a dual value evaluation network structure to evaluate the state action value function. The dual value evaluation network includes a feature value network and a memory value network. A comprehensive reward function is constructed based on the mutual information between features and memories, the distribution distance between source scene features and target scene features, and the memory timeliness measurement. The policy gradient method is used to update the memory operation strategy. Multi-step memory reasoning is performed using the temporal difference method. The memory tracking steps are based on the product of the historical state importance accumulation and the attenuation factor.

[0104] The Actor network encodes the current memory content through a multi-layer perceptron to obtain a memory representation, performs feature transformation on the fused features through a linear transformation to obtain a fused feature representation, and uses a gated recurrent unit network to encode the historical operation sequence to obtain a history representation. The memory representation, fused feature representation, and history representation are concatenated and passed through a fully connected layer to obtain the memory operation strategy distribution. A dual value evaluation network is used to construct the Critic network, including a feature value network based on feature mutual information and distribution distance, and a memory value network based on storage duration and usage frequency. The state-action value function is obtained by combining the outputs of the feature value network and the memory value network through a learnable balance factor.

[0105] A comprehensive reward function based on multi-factor coupling is constructed. The comprehensive reward function includes: calculating the joint distribution and marginal distribution of features and memory through kernel density estimation to obtain a mutual information measurement term, using the maximum mean difference to calculate the distribution distance term of source scene features and target scene features in the reproducing kernel Hilbert space, and modeling the memory timeliness measurement term based on an exponential decay function; designing a grid search-based cross-validation method to determine the weight coefficient of the comprehensive reward function, performing a 5-fold cross-validation within a preset weight search space, using a weighted combination of anomaly detection accuracy, F1 score, and area under the curve as the verification metric, and introducing a dynamic weight adjustment mechanism to adaptively update the initial weights based on the rate of change of mutual information, distribution distance, and memory timeliness;

[0106] The n-step temporal difference method is used to construct the cumulative reward. The reward from the current state to the next n states is combined with the attenuation value of the state action value function. The optimization target is constructed based on the difference between the cumulative reward and the state action value function, and the Actor network parameters are iteratively updated using the policy gradient method. Multi-step memory reasoning is performed based on the temporal difference method. The memory operation strategy is iteratively updated by calculating the temporal difference error between the current state and the target state. A memory tracking mechanism based on importance sampling is introduced to calculate the importance weight of each state in the historical state sequence. The importance weight is the ratio of the current strategy probability to the behavior strategy probability. The cumulative product of the importance weight and the attenuation factor is used as the state importance measure, and the memory operation is tracked based on the state importance.

[0107] For example, a memory reasoning method based on the Actor-Critic framework and a multi-factor coupled reward function is used to improve decision-making efficiency and accuracy in complex scenarios. Its core lies in combining feature information and memory information to make decisions, and optimizing memory operation strategies through multi-step reasoning and memory tracking mechanisms.

[0108] Construct the actor network and the critic network. The actor network is responsible for generating the memory operation policy. It receives the current memory content, fused features, and the historical operation sequence as input. Specifically, the actor network uses a multilayer perceptron to encode the current memory content to obtain a memory representation; uses a linear transformation to transform the fused features to obtain a fused feature representation; and uses a gated recurrent unit network to encode the historical operation sequence to obtain a history representation. These three representations are then concatenated and passed through a fully connected layer to ultimately output the probability distribution of the memory operation policy.

[0109] The Critic network is responsible for evaluating the state-action value function. It adopts a dual value evaluation network structure, including a feature value network and a memory value network. The feature value network is evaluated based on the mutual information between features and memories and the distribution distance between source scene features and target scene features. For example, the kernel density estimation method can be used to calculate the joint distribution and marginal distribution of features and memories to obtain mutual information. The maximum mean difference can be used to calculate the distribution distance between source scene features and target scene features in the reproducing kernel Hilbert space. The memory value network is evaluated based on the storage duration and frequency of use of the memory. For example, the memory value can be calculated based on the length of time the memory is stored and the number of times it is used. Finally, the outputs of the feature value network and the memory value network are combined through a learnable balance factor to obtain the state-action value function.

[0110] A comprehensive reward function is constructed. This function consists of three parts: a mutual information measure between features and memories, a distribution distance measure between source and target scene features, and a memory timeliness measure. The mutual information measure measures the correlation between features and memories. The distribution distance measure measures the difference between source and target scenes. The memory timeliness measure measures the effectiveness of memories. For example, an exponential decay function can be used to model memory timeliness. To determine the weight coefficients of these three terms in the comprehensive reward function, a grid search and five-fold cross-validation method are used. Within the preset weight search space, a five-fold cross-validation method is performed, and the optimal weight coefficient is selected using a weighted combination of anomaly detection accuracy, F1 score, and area under the curve as the verification metric. In addition, a dynamic weight adjustment mechanism is introduced to adaptively update the initial weights based on the rate of change of mutual information, distribution distance, and memory timeliness.

[0111] The cumulative reward is constructed using the n-step temporal difference method. The difference between the cumulative reward and the state-action-value function is used as the optimization target, and the actor network parameters are iteratively updated using the policy gradient method. For example, the cumulative reward can be calculated as the weighted sum of the reward from the current state to the next n states and the decay value of the state-action-value function.

[0112] Multi-step memory reasoning is performed based on the temporal difference method. The memory operation strategy is iteratively updated by calculating the temporal difference error between the current state and the target state.

[0113] A memory tracking mechanism based on importance sampling is introduced. The importance weight of each state in the historical state sequence is calculated as the ratio of the current policy probability to the behavior policy probability. The cumulative product of the importance weight and the decay factor is used as the state importance metric, and memory operations are tracked based on the state importance.

[0114] like Figure 2The figure comparing adaptability after scene changes shows the accuracy recovery trends of three different methods (our method, basic memory network, and spatiotemporal convolutional network (STCN)) as the number of iterations increases after scene changes. The horizontal axis represents the number of iterations (0-800), and the vertical axis represents the detection accuracy (50%-100%). Different line types and dot markers are used in the figure to distinguish the three methods: Our method (solid line + circle markers): Initial accuracy is approximately 95%, which drops to 89% after scene changes, then recovers rapidly, reaching its original level after approximately 500 iterations. Basic memory network (long dashed line + triangle markers): Initial accuracy is approximately 90%, but drops significantly to 71% after scene changes. Recovery is slow, reaching only 87% after 800 iterations. Spatiotemporal convolutional network (short dashed line + square markers): Initial accuracy is approximately 85%, but drops significantly to 57% after scene changes. Recovery is very slow, reaching only 78% after 800 iterations.

[0115] The chart clearly demonstrates three significant advantages of our method after scene changes: 1) minimal performance degradation, only about 6 percentage points; 2) rapid recovery, reaching a stable state in approximately 300-400 iterations; and 3) high recovery, capable of fully recovering to the original performance level. In comparison, the basic memory network experienced a drop of approximately 19 percentage points, and the spatiotemporal convolutional network experienced a drop of approximately 28 percentage points, neither of which fully recovered within 800 iterations. Our method maintains stable detection performance despite scene changes, such as environmental changes, furniture layout adjustments, and new users joining the system, significantly reducing the need for system retraining and providing a more stable and reliable solution for practical applications. Reduced adaptation time and computing resource requirements: The rapid adaptation capability (requiring an average of only 312 iterations) means the system can adapt to new scenes with less computing resources and time, which is particularly important on edge devices with limited computing resources and reduces system downtime. Reduced maintenance costs and user intervention: Traditional methods often require manual intervention or reconfiguration after scene changes. The adaptive capabilities of our method significantly reduce maintenance costs and the need for professional intervention, making the system more suitable for long-term deployment in home environments. Improve the continuity of anomaly detection: During scene transitions, the present invention can still maintain a high accuracy rate (the lowest point is 89%), ensuring the continuity and reliability of anomaly detection and avoiding the problem of large-scale missed or false alarms in traditional methods after environmental changes.

[0116] like Figure 3 The Monte Carlo simulation results are shown in the figure. Figure 3This figure shows the detection accuracy distribution of three different algorithms using the Monte Carlo method, tested on 1000 random scenes. The horizontal axis represents the random scene sample index (0-100), and the vertical axis represents the detection accuracy (60%-100%). The three methods are distinguished by different marker shapes rather than color depth: Our method (circular markers): Detection accuracy is concentrated in the 88.5%-95.3% range, showing a highly concentrated distribution with a standard deviation of only 1.8%. Basic memory networks (triangular markers): Detection accuracy is relatively dispersed between 74.8% and 89.7%, with a standard deviation of 3.9%. Spatiotemporal convolutional networks (STCNs) (square markers): Detection accuracy is highly dispersed between 62.5% and 85.4%, with a standard deviation of 5.7%. The different shapes of the scatter plots visually demonstrate the performance stability of the three methods under various random perturbations. The scatter plots for our method (circular markers) are highly concentrated in the high-accuracy range, demonstrating its strong robustness to various random scene variations. In contrast, the basic memory network (triangle mark) and spatiotemporal convolutional network (square mark) have more dispersed points and lower overall accuracy, indicating that these two methods are more sensitive to scene changes. Furthermore, the minimum accuracy of our method (88.5%) is still higher than the average accuracy of the other two methods, demonstrating its excellent stability. This significantly improves system reliability, reduces the risk of false positives and false negatives, reduces the need for parameter tuning, and improves long-term performance.

[0117] In an optional embodiment,

[0118] The steps of inputting the scenario adaptive features into a multi-task learning network, extracting comprehensive features through a shared representation layer, outputting anomaly prediction results through a task attention layer, inputting the anomaly prediction results into a risk propagation neural network, analyzing the coupling relationship of risk factors through a message passing mechanism, and outputting a risk level include:

[0119] Based on scene-adaptive features, local-to-global features are extracted through the multi-scale convolution kernels of the shared representation layer of the multi-task learning network, and then integrated into a feature pyramid network to obtain comprehensive features. A multi-head collaborative attention mechanism is used to calculate the attention score, and the multi-head outputs are integrated and fused with the comprehensive features through a residual gating mechanism to obtain anomaly prediction results. The anomaly prediction results are input into the risk propagation neural network to construct a risk factor relationship graph. The risk factor nodes and the correlation between factors are processed through the node feature network and the edge feature network respectively, to obtain the edge features of the initial node features and the correlation between risk factors.

[0120] A message passing mechanism based on gated graph convolution is adopted. The message generation submodule combines source node features, target node features, and edge features through a gating unit, and the message aggregation submodule performs weighted aggregation of neighborhood message features based on the attention coefficient between nodes.

[0121] Multiple rounds of message passing are performed on the risk factor relationship graph to iteratively update the node features, and the risk factor node features after a preset number of rounds of message passing are added to the initial features of the risk factor nodes after residual mapping to obtain the risk factor node state; the risk propagation intensity between the risk factor nodes is calculated based on the feature similarity between the risk factor node states and the corresponding edge features, and the risk level is obtained through hierarchical graph pooling integration.

[0122] For example, data preprocessing is performed. Historical data containing various abnormal events and risk factors is collected, and preprocessing operations such as cleaning and normalization are performed on the data. Then, scenario-adaptive features are constructed, selecting appropriate features based on the specific scenario. At the same time, considering the specificity of the scenario, features that can reflect scenario changes are introduced.

[0123] A multi-task learning network is used to extract comprehensive features. The shared representation layer of the network uses convolution kernels of different sizes to extract local and global features, and then performs feature fusion through a feature pyramid network.

[0124] Construct a task attention layer. This layer incorporates a multi-head collaborative attention mechanism, where each attention head independently calculates an attention score and integrates the outputs of multiple attention heads. The integrated features are then fused with the comprehensive features using a residual gating mechanism to produce anomaly prediction results.

[0125] The anomaly prediction results are fed into the risk propagation neural network. First, a risk factor relationship graph is constructed to represent the relationships between risk factors. Then, a node feature network and an edge feature network are used to process the risk factor nodes and the relationships between factors, respectively, to obtain initial node features and edge features. For example, a neural network can be used to extract features for each risk factor and features for the relationships between risk factors.

[0126] A message passing mechanism based on gated graph convolution is designed. The message generation submodule combines source node features, target node features, and edge features to generate message features. The message aggregation submodule performs a weighted summation of message features from neighboring nodes based on the attention coefficient. For example, the features of a risk factor are influenced by its neighboring risk factors, and this influence is simulated through the message passing mechanism.

[0127] Multiple rounds of message passing are performed on the risk factor relationship graph to update the risk factor node features. For example, after multiple rounds of iteration, the features of each risk factor will be updated, thus reflecting the risk propagation process in the network.

[0128] The risk factor node features after iteration are added to the initial features to obtain the risk factor node state. For example, the risk factor node features after multiple rounds of message transmission are added to the initial features to obtain the final risk factor node state.

[0129] The risk propagation strength between risk factor nodes is calculated based on the risk factor node status and edge features. For example, the risk propagation strength is calculated based on the similarity of the risk factor node status and the correlation relationship between them.

[0130] A hierarchical graph pooling operation is used to integrate the node-level risk status and risk propagation intensity to obtain the risk level. For example, the risk factor node status and risk propagation intensity are integrated to obtain a numerical value representing the risk level.

[0131] By combining scenario-adaptive features with multi-task learning, the present invention can more accurately identify abnormal events in different scenarios, thereby improving the accuracy of anomaly detection; by considering the coupling relationship between risk factors and the risk propagation process, the risk level can be assessed more comprehensively to avoid missing potential risk factors; by grading risks, corresponding measures can be taken more specifically, thereby improving the efficiency of risk management.

[0132] In an optional embodiment,

[0133] The steps of inputting the risk level into a recursive neural network, combining historical warning feedback information with feature memory through a gating mechanism, and outputting a warning strategy include:

[0134] The risk level is encoded through a bidirectional long short-term memory network to obtain a fused temporal representation. The historical warning feedback information is input into the feature transformation network to obtain the feedback feature representation. Based on the fused temporal representation and feedback feature representation, the attention weights are calculated at each layer of the time granularity pyramid, and the multi-granularity memory features are obtained through weighted integration through the soft attention mechanism.

[0135] A forget gate and an update gate are designed to dynamically update the multi-granularity memory features, wherein the forget gate controls the degree of retention of historical information based on the fused temporal representation and the multi-granularity memory features, and the update gate controls the proportion of new knowledge introduced, thereby obtaining an updated memory feature. The fused temporal representation and the updated memory feature are input into a gated recurrent unit for adaptive fusion to obtain a warning state representation.

[0136] A multi-objective early warning optimization function is constructed, which comprehensively considers the timeliness of early warning, early warning accuracy and abnormal behavior recognition rate. The early warning state representation is input into the deep Q learning network, and a state-action value function is constructed based on the scene adaptive features and early warning response behavior. The state of the state-action value function includes the scene adaptive features and environmental parameters, and the action of the state-action value function includes the adjustment of the early warning level and the recommendation of intervention measures. The state-action value function is mapped to obtain the early warning strategy, and the early warning strategy is iteratively optimized by the temporal difference algorithm to obtain the optimal early warning strategy.

[0137] For example, the input risk level sequence is encoded. The risk level sequence can be represented as a series of numerical values, such as [1, 2, 3, 4, 5], where larger numerical values ​​represent higher risk levels. This sequence is then input into a bidirectional long short-term memory (BiLSTM) network. The BiLSTM network consists of two LSTM layers, forward and reverse. The forward LSTM layer processes the risk level sequence in chronological order, extracting temporal dependencies within the sequence to produce a series of forward latent state vectors. The reverse LSTM layer processes the risk level sequence in reverse order, extracting reverse temporal dependencies, and producing a series of reverse latent state vectors. The forward and reverse latent state vectors at corresponding moments are concatenated to produce a representation that incorporates bidirectional temporal information, denoted as the fused temporal representation.

[0138] Process historical warning feedback information. This information includes warning time, warning level, response measures, and effect evaluation information. This information is input into a feature transformation network, which can be a multilayer perceptron or convolutional neural network. The feature transformation network converts this information into a fixed-dimensional feature vector, which is recorded as the feedback feature representation. For example, the historical warning feedback information mentioned above may be converted into a 128-dimensional vector after passing through the feature transformation network.

[0139] Construct a time granularity pyramid. The time granularity pyramid can be understood as observing time series at different scales. For example, the risk level sequence can be analyzed at the granularity of days, weeks, and months. At each time granularity layer, the attention weight is calculated based on the fusion of time series representation and feedback feature representation. For example, at the "day" granularity layer, the correlation between the risk level of each day and the historical warning feedback information is calculated to obtain the attention weight of each day. The attention weight reflects the importance of different time points. Through the soft attention mechanism, the attention features of different time granularities are weighted and integrated to obtain multi-granularity memory features. For example, the attention features of the three granularities of "day", "week", and "month" are weighted and summed to obtain the final multi-granularity memory feature.

[0140] To enable the model to dynamically update its memory, a forget gate and an update gate are designed to dynamically update the multi-granularity memory features. The forget gate controls the degree of retention of historical information based on the fusion of temporal representation and multi-granularity memory features. For example, if the current risk level is high, the forget gate may reduce the weight of historical information. The update gate controls the proportion of new knowledge introduced. For example, if the current warning feedback information is very important, the update gate may increase the weight of new information. Through the forget gate and update gate, the updated memory features are obtained.

[0141] The fused temporal representation and updated memory features are fed into a gated recurrent unit (GRU) for adaptive fusion to obtain a warning state representation. The GRU can effectively fuse information from different sources and capture long-term dependencies in the sequence.

[0142] A multi-objective early warning optimization function is constructed. This function comprehensively considers early warning timeliness, accuracy, and abnormal behavior recognition rate. The early warning state representation is input into a deep Q-learning network. The deep Q-learning network constructs a state-action value function based on the scenario-adaptive features and the early warning response behavior. The state of the state-action value function includes the scenario-adaptive features and environmental parameters, such as the current risk level and the number of historical early warnings. Actions include adjusting the early warning level and recommending intervention measures, such as raising the early warning level from level 1 to level 2 or recommending certain emergency measures. The deep Q-learning network maps the state-action value function to an early warning strategy. The early warning strategy is iteratively optimized using a temporal difference algorithm to obtain the optimal early warning strategy. For example, if an early warning strategy can provide early warning and effectively reduce risk, the value of the strategy is enhanced.

[0143] Traditional time series analysis primarily uses unidirectional LSTMs or simple RNNs to process time series data, which cannot fully capture the bidirectional evolution of risk sequences. Considering that risk sequences contain both historical cumulative effects and future development trends, and that risks exhibit different evolutionary characteristics at different time scales, this paper proposes an early warning method based on a bidirectional long-short-term memory network and multi-granularity time series analysis. First, a BiLSTM architecture is employed to extract temporal dependencies through forward and backward LSTM layers, respectively, enabling bidirectional analysis of risk sequences. Second, a feature transformation network is designed to convert historical information, such as warning time, level, measures, and effects, into a unified feature representation. An innovative time granularity pyramid is constructed to analyze risk evolution characteristics at different time scales, such as days, weeks, and months, adaptively fusing multi-scale features through a soft attention mechanism. A dynamic update mechanism based on forget gates and update gates is designed to adaptively control the retention of historical information and the introduction of new knowledge by assessing the current risk level and the importance of historical information. This mechanism enables the system to rapidly respond to new risk situations while retaining important historical experience. Deep Q-learning is employed to construct a state-action value function, transforming the early warning problem into a multi-objective optimization problem. By considering scenario characteristics and environmental parameters, the system can adaptively adjust the warning level and recommend intervention measures, and continuously optimize the warning strategy through the temporal difference algorithm.

[0144] A second aspect of an embodiment of the present invention provides a home abnormal state signal detection system based on multimodal sensing, the system comprising:

[0145] The first unit is used to input the collected human motion data, acoustic features and environmental parameters into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatiotemporal attention mechanism, extracts spatiotemporal feature sequences through a multi-scale convolutional neural network, and combines them with the spectral analysis results of the acoustic features to form a joint probability feature;

[0146] The second unit is used to process the joint probability features through the temporal knowledge graph of the graph attention network to obtain multidimensional combined features, input the multidimensional combined features into the two-stream autoencoder network, perform feature mapping on the source scene and use adversarial training to obtain target scene features; input the source scene features and target scene features into the memory-enhanced neural network, and output scene-adaptive features in combination with the reinforcement learning method;

[0147] The third unit is used to input the scenario adaptive features into the multi-task learning network, extract comprehensive features through the shared representation layer, output the anomaly prediction results through the task attention layer, input the anomaly prediction results into the risk propagation neural network, analyze the coupling relationship of risk factors through the message passing mechanism and output the risk level; input the risk level into the recursive neural network, combine the historical warning feedback information with the feature memory through the gating mechanism and output the warning strategy.

[0148] According to a third aspect of the embodiments of the present invention,

[0149] An electronic device is provided, comprising:

[0150] processor;

[0151] a memory for storing processor-executable instructions;

[0152] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0153] According to a fourth aspect of the embodiments of the present invention,

[0154] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0155] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal home status signals based on multimodal sensing, characterized in that: include: The collected human motion data, acoustic features and environmental parameters are input into a deep fusion network, which dynamically assigns weights to different features through a spatiotemporal attention mechanism, extracts spatiotemporal feature sequences through a multi-scale convolutional neural network, and combines them with the spectrum analysis results of acoustic features to form joint probability features, including: filtering, transforming and normalizing the human motion data, acoustic signals and environmental parameters to obtain standardized features; passing the standardized features in turn through the time attention layer and modal attention layer of a two-layer attention network to obtain weighted multimodal features; inputting the multimodal features into a multi-branch convolutional neural network, and extracting the spatiotemporal feature sequence through deep separable convolution and residual connection; inputting the spatiotemporal feature sequence into a Gaussian mixture model, which includes multiple Gaussian components, each of which has a corresponding mixing weight , mean vector and covariance matrix, the mean vector and covariance matrix are functions of latent variables, and the latent variables are recursively updated through the long short-term memory network to obtain the spatiotemporal feature probability distribution; the spatiotemporal feature probability distribution and acoustic features are input into the conditional variational autoencoder, and the conditional variational autoencoder includes a two-branch encoder network, and the conditional variational autoencoder obtains high-level feature representation through the two-branch encoder network, and outputs the posterior distribution parameters through the mean network and the variance network; the posterior distribution parameters are reparameterized and sampled and reconstructed through the decoder network, and the evidence lower bound of the reconstruction term and the KL divergence is calculated; the gradient norm of the evidence lower bound on the spatiotemporal feature probability distribution and acoustic features is calculated to obtain the modal importance score, the KL divergence weight is dynamically adjusted and the evidence lower bound is maximized, and the conditional variational autoencoder is trained to obtain the joint probability feature of action and sound; The joint probability features are processed by the temporal knowledge graph of the graph attention network to obtain multi-dimensional combined features, the multi-dimensional combined features are input into the dual-stream autoencoder network, the source scene is feature mapped and the target scene features are obtained by adversarial training; the source scene features and the target scene features are input into the memory enhancement neural network, and the scene adaptive features are output in combination with the reinforcement learning method, including: inputting the source scene features and the target scene features into the hierarchical memory writing mechanism of the memory enhancement neural network for feature normalization processing, interacting through the multi-head attention mechanism to obtain fused features, determining the target memory slot based on local sensitive hashing, and writing the fused features into the memory module through the gated writing mechanism; the memory module is input into the reinforcement learning framework, and the reinforcement learning framework includes the Actor-Critic network structure and the multi-step memory reasoning mechanism, Ac The tor network generates a memory operation strategy based on the current memory content, the fusion features and the historical operation sequence. The critic network adopts a dual value evaluation network structure to evaluate the state action value function. It constructs a reward function based on feature mutual information, distribution distance and memory timeliness, adopts a policy gradient update strategy and performs multi-step memory reasoning through temporal difference. A multi-scale query vector is constructed for the fusion features. The multi-scale query vector includes a spatial scale vector, a time scale vector and a channel scale vector. The memory feature with the maximum cosine similarity to the query vector is retrieved from the memory module, integrated with the fusion feature through an adaptive fusion network, and a scene adaptive feature is obtained through a residual connection. The memory module is optimized based on a weighted combination of memory duration and feature mutual information, including priority update, memory compression and dynamic adjustment of memory capacity. The scenario adaptive features are input into the multi-task learning network, the comprehensive features are extracted through the shared representation layer, the anomaly prediction results are output through the task attention layer, the anomaly prediction results are input into the risk propagation neural network, the risk factor coupling relationship is analyzed through the message passing mechanism and the risk level is output, including: based on the scenario adaptive features, the local to global features are extracted through the multi-scale convolution kernel of the shared representation layer of the multi-task learning network, and the comprehensive features are obtained by fusion through the feature pyramid network; the attention score is calculated using the multi-head collaborative attention mechanism, the multi-head outputs are integrated and then fused with the comprehensive features through the residual gating mechanism to obtain the anomaly prediction results; the anomaly prediction results are input into the risk propagation neural network, and a risk factor relationship graph is constructed. The risk factor nodes and the correlation relationship between factors are processed respectively through the node feature network and the edge feature network to obtain the initial node features and the edge features of the correlation relationship between risk factors; the message passing mechanism of the gated graph convolution is adopted, wherein the message generation submodule combines the source node features, the target node features and the edge features through the gating unit, and the message aggregation submodule performs weighted aggregation on the neighborhood message features based on the attention coefficient between nodes; the node features are updated through multiple rounds of message passing on the risk factor relationship graph, and the node features are updated through the message passing mechanism. The risk factor node features after a preset number of rounds of message transmission are added to the initial features of the risk factor node after residual mapping to obtain the risk factor node state; the risk propagation intensity between the risk factor nodes is calculated based on the feature similarity between the risk factor node states and the corresponding edge features, and the risk level is obtained through hierarchical graph pooling integration; the risk level is input into a recursive neural network, and the feature memory is performed through a gating mechanism in combination with historical warning feedback information to output a warning strategy, including: encoding the risk level through a bidirectional long short-term memory network to obtain a fused temporal representation; inputting the historical warning feedback information into a feature transformation network to obtain a feedback feature representation; calculating the attention weights at each layer of the time granularity pyramid based on the fused temporal representation and the feedback feature representation, and obtaining a multi-granularity memory feature through weighted integration through a soft attention mechanism; designing a forget gate and an update gate to dynamically update the multi-granularity memory feature, wherein the forget gate controls the degree of retention of historical information based on the fused temporal representation and the multi-granularity memory feature, and the update gate controls the introduction ratio of new knowledge to obtain an updated memory feature; the fused temporal representation and the updated memory feature are input into a gated recurrent unit for adaptive fusion to obtain a warning state representation;A multi-objective early warning optimization function is constructed, which comprehensively considers early warning timeliness, early warning accuracy, and abnormal behavior recognition rate. The early warning state representation is input into a deep Q-learning network. A state-action value function is constructed based on the scene adaptive features and early warning response behavior. The state of the state-action value function includes the scene adaptive features and environmental parameters, and the actions of the state-action value function include early warning level adjustment and intervention measure recommendation. The state-action value function is mapped to a warning strategy, and the warning strategy is iteratively optimized using a temporal difference algorithm to obtain the optimal warning strategy.

2. The method according to claim 1, characterized in that The steps of processing the joint probability features through the temporal knowledge graph of the graph attention network to obtain multidimensional combined features, inputting the multidimensional combined features into the two-stream autoencoder network, performing feature mapping on the source scene, and obtaining target scene features through adversarial training include: The joint probability features are input into the time series knowledge graph, and the correlation weights between time series nodes are calculated through the attention mechanism. The time series nodes correspond to feature vectors at different moments. The weighted time series knowledge graph is processed by the multi-head attention mechanism of the multi-layer graph attention network, and multi-dimensional combined features are obtained by multi-layer stacking. Inputting the multidimensional combined features into a two-stream autoencoder network, wherein the source scene encoder branch is composed of a multi-layer perceptron, encoding the input features into source scene latent features through layer-by-layer nonlinear transformation, and the target scene adapter branch introduces a residual connection in the encoder, adding the source scene latent features and the encoder output features of the target scene adapter branch to obtain the target scene latent features, and reconstructing the target scene features through the decoder network; Inputting the target scene features into a discriminant network with a block discriminant structure, performing local discrimination on the features through multi-layer convolution, calculating the Wasserstein distance between the source scene potential features and the target scene potential features as the adversarial loss and introducing a gradient penalty term; The maximum average difference between the latent features of the source scene and the target scene is calculated as a distribution alignment constraint, and reconstruction constraints, cycle consistency constraints and contrastive learning losses are introduced at the same time; The adversarial loss of the discriminator network and the joint loss of the target scene adapter are optimized alternately by exponential sliding average. The joint loss includes adversarial loss, distribution alignment loss, cycle consistency loss and contrastive loss.

3. The method according to claim 1, characterized in that The memory module is input into a reinforcement learning framework, which includes an Actor-Critic network structure and a multi-step memory reasoning mechanism. The Actor network generates a memory operation strategy based on the current memory content, the fusion features, and the historical operation sequence. The Critic network uses a dual value evaluation network structure to evaluate the state action value function. A reward function is constructed based on feature mutual information, distribution distance, and memory timeliness. The policy gradient update strategy is used and multi-step memory reasoning is performed through temporal difference. The steps include: The Actor network encodes the current memory content through a multi-layer perceptron to obtain a memory representation, performs feature transformation on the fused features through a linear transformation to obtain a fused feature representation, and uses a gated recurrent unit network to encode the historical operation sequence to obtain a history representation. The memory representation, fused feature representation, and history representation are concatenated and passed through a fully connected layer to obtain the memory operation strategy distribution. A dual value evaluation network is used to construct the Critic network, including a feature value network based on feature mutual information and distribution distance, and a memory value network based on storage duration and usage frequency. The state-action value function is obtained by combining the outputs of the feature value network and the memory value network through a learnable balance factor. Construct a comprehensive reward function based on multi-factor coupling, including a mutual information measurement term between features and memory, a distribution distance term between source and target scene features, and a memory timeliness measurement term. The weight coefficients are determined through grid search and cross-validation, and the weights are dynamically adjusted based on the mutual information change rate, distribution distance change rate, and memory timeliness change rate. Cumulative rewards are constructed using n-step temporal differences, and the Actor network is updated based on the rewards from the current to the next n states and the decay value of the state-action-value function. The memory operation strategy is iteratively updated through temporal difference errors. A memory tracking mechanism based on importance sampling is introduced to calculate the importance weight of each state in the historical state sequence. The importance weight is the ratio of the current strategy probability to the behavior strategy probability. The cumulative product of the importance weight and the decay factor is used as the state importance measure, and memory operations are tracked based on the state importance.

4. A home abnormal state signal detection system based on multimodal sensing, used to implement the method according to any one of claims 1 to 3, characterized in that: include: The first unit is used to input the collected human motion data, acoustic features and environmental parameters into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatiotemporal attention mechanism, extracts spatiotemporal feature sequences through a multi-scale convolutional neural network, and combines them with the spectral analysis results of the acoustic features to form a joint probability feature; The second unit is used to process the joint probability features through the temporal knowledge graph of the graph attention network to obtain multi-dimensional combined features, input the multi-dimensional combined features into the two-stream autoencoder network, perform feature mapping on the source scene, and use adversarial training to obtain the target scene features; The source scene features and target scene features are input into the memory-enhanced neural network, and the scene adaptive features are output in combination with the reinforcement learning method. The third unit is used to input the scenario adaptive features into the multi-task learning network, extract comprehensive features through the shared representation layer, output the anomaly prediction results through the task attention layer, input the anomaly prediction results into the risk propagation neural network, analyze the coupling relationship of risk factors through the message passing mechanism and output the risk level; input the risk level into the recursive neural network, combine the historical warning feedback information with the feature memory through the gating mechanism and output the warning strategy.

5. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images