Home abnormal state signal detection method and system based on multi-mode sensing

Abstract

The invention provides a home abnormal state signal detection method and system based on multi-modal sensing, and relates to the technical field of detection, and the method comprises the steps: collecting human body motion, acoustics and environment parameters through a millimeter wave radar, an acoustic sensor and an environment sensor, inputting the parameters into a deep fusion network, a spatio-temporal attention mechanism and a multi-scale convolutional neural network are used to extract a spatio-temporal feature sequence, and joint probability features are formed in combination with an acoustic feature spectrum analysis result. And inputting the joint probability features into a time sequence knowledge graph, and obtaining scene adaptive features after processing by a graph attention network and a double-flow auto-encoder network. And finally, the multi-task learning network is combined with the risk propagation neural network and the recursive neural network to carry out anomaly prediction and risk level evaluation, and is combined with historical early warning information to output an early warning strategy. According to the invention, multi-modal data can be effectively fused, the accuracy and reliability of home abnormal state detection are improved, and a more accurate risk assessment and early warning strategy is provided.

Description

Technical Field

[0001] The present invention relates to detection technologies, and in particular, to a method and system for detecting abnormal state signals at home based on multi-modal sensing. Background Art

[0002] The technology for detecting abnormal states at home aims to sense abnormal events in the home environment through sensors, such as falls, fires, intrusions, etc., and issue alarms in a timely manner to ensure the safety of groups such as the elderly and children, as well as the safety of household property. Existing technologies for detecting abnormal states at home mainly rely on single sensors or simple multi-sensor fusion methods, such as relying only on camera image recognition or only on sound recognition. Such methods have the following deficiencies:

[0003] Poor robustness: Single sensors are vulnerable to environmental interference, such as changes in light, noise, etc., resulting in a decrease in detection accuracy. Simple multi-sensor fusion methods also fail to fully utilize the complementary information between different sensors, resulting in poor detection effects in complex scenarios.

[0004] Lack of scene adaptability: Existing methods are usually based on pre-set rules or models and are difficult to adapt to the differences in different home environments, such as furniture layout, people's activity habits, etc., resulting in a high false alarm rate.

[0005] Insufficient risk assessment ability: Existing methods usually only focus on the detection of abnormal events and lack the assessment of risk levels. For example, the same fall event poses different risk levels to the elderly and the young. The lack of risk assessment ability will lead to the inability to effectively take corresponding intervention measures. Summary of the Invention

[0006] Embodiments of the present invention provide a method and system for detecting abnormal state signals at home based on multi-modal sensing, which can solve the problems in the prior art.

[0007] In the first aspect of the embodiments of the present invention,

[0008] A method for detecting abnormal state signals at home based on multi-modal sensing is provided, including:

[0009] Inputting the collected human motion data, acoustic features, and environmental parameters into a deep fusion network, where the deep fusion network dynamically assigns weights to different features through a spatio-temporal attention mechanism, extracts a spatio-temporal feature sequence through a multi-scale convolutional neural network, and combines it with the spectral analysis result of the acoustic features to form a joint probability feature;

[0010] The combined probability features are processed by the temporal knowledge graph of the graph attention network to obtain multi-dimensional combined features. The multi-dimensional combined features are input into the two-stream autoencoder network to perform feature mapping on the source scenario and obtain the target scenario features using adversarial training. The source scenario features and the target scenario features are input into the memory-augmented neural network, and the scene adaptive features are output by combining the reinforcement learning method.

[0011] The scene adaptive features are input into the multi-task learning network. The comprehensive features are extracted through the shared representation layer, and the anomaly prediction results are output through the task attention layer. The anomaly prediction results are input into the risk propagation neural network, and the coupling relationship of risk factors is analyzed through the message passing mechanism and the risk level is output. The risk level is input into the recurrent neural network, and the feature memory is performed through the gating mechanism in combination with the historical warning feedback information and the warning strategy is output.

[0012] In an alternative embodiment,

[0013] The steps of inputting the collected human motion data, acoustic features, and environmental parameters into the deep fusion network to form combined probability features include:

[0014] The human motion data, acoustic signals, and environmental parameters are filtered, transformed, and normalized to obtain standardized features. The standardized features are sequentially passed through the temporal attention layer and the modality attention layer of the double-layer attention network to obtain weighted multi-modal features. The multi-modal features are input into the multi-branch convolutional neural network, and the spatio-temporal feature sequence is extracted through depthwise separable convolution and residual connection.

[0015] The spatio-temporal feature sequence is input into the Gaussian mixture model. The Gaussian mixture model includes multiple Gaussian components, each Gaussian component has a corresponding mixing weight, mean vector, and covariance matrix. The mean vector and covariance matrix are both functions of the latent variables. The latent variables are recursively updated through the long short-term memory network to obtain the spatio-temporal feature probability distribution. The spatio-temporal feature probability distribution and the acoustic features are input into the conditional variational autoencoder. The conditional variational autoencoder includes a two-branch encoder network. The conditional variational autoencoder obtains the high-level feature representation through the two-branch encoder network, and outputs the posterior distribution parameters through the mean network and the variance network. The posterior distribution parameters are reparameterized and sampled and reconstructed through the decoder network, and the evidence lower bound of the reconstruction term and the KL divergence is calculated. The gradient norm of the evidence lower bound with respect to the spatio-temporal feature probability distribution and the acoustic features is calculated to obtain the modality importance score, the KL divergence weight is dynamically adjusted, and the evidence lower bound is maximized. The conditional variational autoencoder is trained to obtain the joint probability features of actions and sounds.

[0016] In an alternative embodiment,

[0017] The step of obtaining multi-dimensional combined features by processing the joint probability features through the temporal knowledge graph of the graph attention network and inputting the multi-dimensional combined features into the dual-stream autoencoder network to perform feature mapping on the source scene and obtaining the target scene features using adversarial training includes:

[0018] Input the joint probability features into the temporal knowledge graph, calculate the correlation weights between temporal nodes through the attention mechanism, where the temporal nodes correspond to feature vectors at different moments; process the weighted temporal knowledge graph through the multi-head attention mechanism of the multi-layer graph attention network, and obtain multi-dimensional combined features through multi-layer stacking;

[0019] Input the multi-dimensional combined features into the dual-stream autoencoder network, where the source scene encoder branch is composed of multi-layer perceptrons, and the input features are encoded into source scene latent features through layer-by-layer non-linear transformation. The target scene adapter branch introduces residual connections in the encoder, adds the source scene latent features and the encoder output features of the target scene adapter branch to obtain the target scene latent features, and reconstructs the target scene features through the decoder network;

[0020] Input the target scene features into the discriminator network with a block discriminant structure, perform local discrimination on the features through multi-layer convolution, calculate the Wasserstein distance between the source scene latent features and the target scene latent features as the adversarial loss and introduce a gradient penalty term;

[0021] Calculate the maximum mean difference between the source scene and the target scene latent features as the distribution alignment constraint, and at the same time introduce the reconstruction constraint, the cycle consistency constraint and the contrast learning loss;

[0022] Alternately optimize the adversarial loss of the discriminator network and the joint loss of the target scene adapter through exponential moving average. The joint loss includes the adversarial loss, the distribution alignment loss, the cycle consistency loss and the contrast loss.

[0023] In an alternative implementation,

[0024] The step of inputting the source scene features and the target scene features into the memory-augmented neural network and outputting the scene adaptive features in combination with the reinforcement learning method includes:

[0025] Input the source scene features and the target scene features into the hierarchical memory writing mechanism of the memory-augmented neural network for feature normalization processing, perform interaction through the multi-head attention mechanism to obtain the fusion features, determine the target memory slot based on local sensitive hashing, and write the fusion features into the memory module through the gated writing mechanism;

[0026] Input the memory module into a reinforcement learning framework, which includes an Actor-Critic network structure and a multi-step memory reasoning mechanism. The Actor network generates a memory operation policy based on the current memory content, the fusion feature, and the historical operation sequence. The Critic network uses a dual-value evaluation network structure to evaluate the state-action value function, constructs a reward function based on feature mutual information, distribution distance, and memory timeliness, updates the policy using policy gradients, and performs multi-step memory reasoning through temporal difference.

[0027] Construct a multi-scale query vector for the fusion feature. The multi-scale query vector includes a spatial scale vector, a temporal scale vector, and a channel scale vector. Retrieve the memory feature with the maximum cosine similarity to the query vector from the memory module, integrate it with the fusion feature through an adaptive fusion network, and obtain the scene adaptive feature through a residual connection.

[0028] Optimize the memory module based on the weighted combination of memory duration and feature mutual information, including priority update, memory compression, and dynamic adjustment of memory capacity.

[0029] In an alternative implementation

[0030] Input the memory module into a reinforcement learning framework, which includes an Actor-Critic network structure and a multi-step memory reasoning mechanism. The Actor network generates a memory operation policy based on the current memory content, the fusion feature, and the historical operation sequence. The Critic network uses a dual-value evaluation network structure to evaluate the state-action value function, constructs a reward function based on feature mutual information, distribution distance, and memory timeliness, updates the policy using policy gradients, and performs multi-step memory reasoning. The steps include:

[0031] The Actor network encodes the current memory content through a multi-layer perceptron to obtain a memory representation, performs a feature transformation on the fusion feature through a linear transformation to obtain a fusion feature representation, encodes the historical operation sequence through a gated recurrent unit network to obtain a historical representation, concatenates the memory representation, the fusion feature representation, and the historical representation, and obtains a memory operation policy distribution through a fully connected layer; constructs the Critic network using a dual-value evaluation network, including a feature value network based on feature mutual information and distribution distance, and a memory value network based on storage duration and usage frequency, and obtains the state-action value function by combining the outputs of the feature value network and the memory value network through a learnable balance factor.

[0032] Construct a comprehensive reward function based on multi-factor coupling, including the mutual information measurement term between features and memories, the distribution distance term between source scenario and target scenario features, and the memory timeliness measurement term; assume that the weight coefficients are determined by grid search and cross-validation, and the weights are dynamically adjusted according to the change rates of mutual information, distribution distance, and memory timeliness;

[0033] Use n-step temporal difference to construct the cumulative reward, and update the Actor network based on the rewards from the current to the next n states and the decay values of the state-action value function; iteratively update the memory operation strategy through the temporal difference error; introduce a memory tracking mechanism based on importance sampling to calculate the importance weights of each state in the historical state sequence, where the importance weight is the ratio of the current policy probability to the behavior policy probability, and use the cumulative product of the importance weight and the decay factor as the state importance measure to track the memory operation based on the state importance.

[0034] In an alternative implementation,

[0035] The steps of inputting the scene adaptive features into the multi-task learning network, extracting comprehensive features through the shared representation layer, outputting the anomaly prediction results through the task attention layer, and inputting the anomaly prediction results into the risk propagation neural network to analyze the coupling relationship of risk factors and output the risk level by the message passing mechanism include:

[0036] Based on the scene adaptive features, extract local-to-global features through the multi-scale convolutional kernels of the shared representation layer of the multi-task learning network, and fuse them through the feature pyramid network to obtain comprehensive features; calculate the attention scores using the multi-head collaborative attention mechanism, integrate the multi-head outputs, and fuse them with the comprehensive features through the residual gating mechanism to obtain the anomaly prediction results; input the anomaly prediction results into the risk propagation neural network, construct a risk factor relationship graph, and process the risk factor nodes and the association relationships between factors through the node feature network and the edge feature network respectively to obtain the initial node features and the edge features of the association relationships between risk factors;

[0037] Adopt the message passing mechanism of gated graph convolution, where the message generation sub-module combines the source node features, target node features, and edge features through a gated unit, and the message aggregation sub-module weights and aggregates the neighborhood message features based on the attention coefficients between nodes;

[0038] Perform multiple rounds of message passing iterations on the risk factor relationship graph to update the node features, add the risk factor node features after a preset number of rounds of message passing to the initial risk factor node features after residual mapping to obtain the risk factor node states; calculate the risk propagation intensity between risk factor nodes based on the feature similarity between risk factor node states and the corresponding edge features, and integrate them through hierarchical graph pooling to obtain the risk level.

[0039] In an alternative embodiment,

[0040] The steps of inputting the risk level into a recurrent neural network and performing feature memory and outputting an early warning strategy through a gating mechanism in combination with historical early warning feedback information include:

[0041] Encoding the risk level through a bidirectional long short-term memory network to obtain a fused time series representation; inputting the historical early warning feedback information into a feature transformation network to obtain a feedback feature representation; calculating attention weights at each layer of the time granularity pyramid based on the fused time series representation and the feedback feature representation, and weighted integration through a soft attention mechanism to obtain multi-granularity memory features;

[0042] Designing a forgetting gate and an update gate to dynamically update the multi-granularity memory features, where the forgetting gate controls the retention degree of historical information based on the fused time series representation and the multi-granularity memory features, and the update gate controls the introduction ratio of new knowledge to obtain updated memory features; inputting the fused time series representation and the updated memory features into a gated recurrent unit for adaptive fusion to obtain an early warning state representation;

[0043] Constructing a multi-objective early warning optimization function, which comprehensively considers early warning timeliness, early warning accuracy, and abnormal behavior recognition rate. Inputting the early warning state representation into a deep Q-learning network, constructing a state-action value function based on scenario adaptive features and early warning response behaviors. The state of the state-action value function includes scenario adaptive features and environmental parameters, and the actions of the state-action value function include early warning level adjustment and intervention measure recommendation. Mapping the state-action value function to obtain an early warning strategy, and iteratively optimizing the early warning strategy through a temporal difference algorithm to obtain an optimal early warning strategy.

[0044] In the second aspect of the embodiments of the present invention,

[0045] A home abnormal state signal detection system based on multi-modal sensing is provided, including:

[0046] A first unit for inputting the collected human motion data, acoustic features, and environmental parameters into a deep fusion network. The deep fusion network performs dynamic weight allocation on different features through a spatio-temporal attention mechanism, extracts a spatio-temporal feature sequence through a multi-scale convolutional neural network, and combines it with the spectral analysis result of the acoustic features to form a joint probability feature;

[0047] A second unit for processing the joint probability feature through the temporal knowledge graph of a graph attention network to obtain a multi-dimensional combined feature, inputting the multi-dimensional combined feature into a two-stream autoencoder network, performing feature mapping on the source scene and obtaining target scene features through adversarial training; inputting the source scene features and the target scene features into a memory-enhanced neural network, and outputting scenario adaptive features in combination with a reinforcement learning method;

[0048] A third unit is configured to input the scene adaptive features into a multi-task learning network, extract comprehensive features through a shared representation layer, output an anomaly prediction result through a task attention layer, input the anomaly prediction result into a risk propagation neural network, analyze the coupling relationship of risk factors through a message passing mechanism and output a risk level; input the risk level into a recurrent neural network, perform feature memory through a gating mechanism in combination with historical warning feedback information and output a warning strategy.

[0049] In a third aspect of the embodiments of the present invention,

[0050] there is provided an electronic device, comprising:

[0051] a processor;

[0052] a memory for storing instructions executable by the processor;

[0053] wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0054] In a fourth aspect of the embodiments of the present invention,

[0055] there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0056] The present invention fuses human motion, acoustic features and environmental parameters through multi-modal sensing, and uses a deep learning network for feature extraction and analysis, which can more comprehensively capture the signals of abnormal states, thereby improving the detection accuracy and reducing the false alarm rate and missed alarm rate.

[0057] The present invention adopts a two-stream autoencoder network and a memory-augmented neural network, which can be adaptively adjusted according to different scenarios, and combines a risk propagation neural network and a recurrent neural network for risk level evaluation and warning strategy generation, realizing more accurate risk warning.

[0058] The present invention utilizes a temporal knowledge graph and a multi-task learning network, which can better capture the spatio-temporal correlation and potential risk factors between data, and enhance the interpretability of the system through a graph attention network and a message passing mechanism, improving the robustness and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic flowchart of a method for detecting abnormal state signals at home based on multi-modal sensing according to an embodiment of the present invention;

[0060] Figure 2 It is a comparison chart of the adaptability after scene conversion;

[0061] Figure 3Monte Carlo simulation result graph. Detailed implementation manners

[0062] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0063] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0064] Figure 1 It is a flowchart of a method for detecting abnormal state signals at home based on multi-modal sensing according to an embodiment of the present invention. As Figure 1 shown, the method includes:

[0065] Collect human motion data through a millimeter-wave radar, collect acoustic features through an acoustic sensor, and collect environmental parameters through a temperature, humidity and air quality sensor; input the human motion data, acoustic features and environmental parameters into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatio-temporal attention mechanism, and inputs the weight assignment result into a multi-scale convolutional neural network to extract a spatio-temporal feature sequence. The spatio-temporal feature sequence and the spectral analysis result of the acoustic features are combined to form a joint probability feature;

[0066] Input the joint probability feature into a temporal knowledge graph. The temporal knowledge graph performs feature interaction operations through a graph attention network and outputs a multi-dimensional combined feature. Input the multi-dimensional combined feature into a two-stream autoencoder network. The first branch of the two-stream autoencoder network performs feature mapping on the source scene, and the second branch calculates the target scene feature through adversarial training; input the source scene feature and the target scene feature into a memory-augmented neural network. The memory-augmented neural network combines a reinforcement learning method to output a scene-adaptive feature;

[0067] Input the scene-adaptive feature into a multi-task learning network. The multi-task learning network extracts comprehensive features through a shared representation layer and outputs an abnormal prediction result through a task attention layer. Input the abnormal prediction result into a risk propagation neural network for risk propagation analysis, analyze the coupling relationship of risk factors through a message passing mechanism and output a risk level; input the risk level into a recurrent neural network. The recurrent neural network combines historical warning feedback information to perform feature memory through a gating mechanism and outputs a warning strategy.

[0068] In an alternative embodiment,

[0069] The steps of inputting the collected human motion data, acoustic features, and environmental parameters into the deep fusion network to form joint probability features include:

[0070] Filter, transform, and normalize the human motion data, acoustic signals, and environmental parameters to obtain standardized features; sequentially pass the standardized features through the temporal attention layer and the modal attention layer of the double-layer attention network to obtain weighted multi-modal features; input the multi-modal features into the multi-branch convolutional neural network, and extract the spatio-temporal feature sequence through depthwise separable convolution and residual connection;

[0071] Input the spatio-temporal feature sequence into the Gaussian mixture model. The Gaussian mixture model includes multiple Gaussian components, each Gaussian component having a corresponding mixture weight, mean vector, and covariance matrix. The mean vector and covariance matrix are both functions of latent variables, and the latent variables are recursively updated through a long short-term memory network to obtain the spatio-temporal feature probability distribution; input the spatio-temporal feature probability distribution and acoustic features into the conditional variational autoencoder. The conditional variational autoencoder includes a two-branch encoder network. The conditional variational autoencoder obtains the high-level feature representation through the two-branch encoder network, and outputs the posterior distribution parameters after passing through the mean network and the variance network; perform reparameterization sampling on the posterior distribution parameters and reconstruct through the decoder network, and calculate the evidence lower bound of the reconstruction term and the KL divergence; calculate the gradient norm of the evidence lower bound with respect to the spatio-temporal feature probability distribution and acoustic features to obtain the modal importance score, dynamically adjust the KL divergence weight, and maximize the evidence lower bound to train the conditional variational autoencoder to obtain the joint probability features of actions and sounds.

[0072] Exemplarily, preprocess the collected multi-modal data. The human motion data can come from a depth sensor or an inertial sensor, and is converted into a three-dimensional joint coordinate sequence. The acoustic signals are collected using a microphone, and the environmental parameters can be obtained using a temperature sensor, a humidity sensor, etc. Filter these data of different modalities, for example, use Kalman filtering to remove noise, and then perform data transformation, for example, perform short-time Fourier transform on the acoustic signals to obtain a spectrogram. Finally, normalize the data of all modalities to the same numerical range, for example, between 0 and 1, to form standardized features. For example, the human motion data can be represented as the three-dimensional coordinates of each joint at each time point, the acoustic signals can be represented as the spectral energy of each time frame, and the environmental parameters can be represented as temperature and humidity values.

[0073] Dynamic weight allocation is performed on the normalized features using a two-layer attention network. The two-layer attention network includes a temporal attention layer and a modal attention layer. The temporal attention layer is used to learn the weights of different time steps. For example, higher weights are assigned to the start and end stages of an action. The modal attention layer is used to learn the weights of different modalities. For example, when acoustic information is more important, higher weights are assigned to acoustic features. The temporal attention mechanism and the modal attention mechanism can be implemented in various ways. For example, an attention mechanism based on scaled dot product can be used. Suppose there are three modalities: motion, sound, and environment. The temporal attention layer can output a weight vector of length equal to the number of time steps, and the modal attention layer can output a weight vector of length 3. Multiply these two weight vectors to obtain the weights of each modality at each time step. Apply these weights to the normalized features to obtain weighted multi-modal features.

[0074] The weighted multi-modal features are input into a multi-branch convolutional neural network to extract spatio-temporal feature sequences. The multi-branch convolutional neural network uses convolutional kernels of different sizes to capture spatio-temporal features at different scales. For example, small-sized convolutional kernels are used to capture local detail features, and large-sized convolutional kernels are used to capture global context information. Depthwise separable convolutions can reduce the computational amount, and residual connections can avoid the problem of gradient vanishing, thereby improving the training efficiency and performance of the network. The output of the multi-branch convolutional neural network is a spatio-temporal feature sequence that contains feature information at different scales and different modalities. Suppose the input feature dimension is 128 and the time step is 64, then the dimension of the output spatio-temporal feature sequence can be 256 and the time step is 32.

[0075] The spatio-temporal feature sequence is input into a Gaussian mixture model (GMM) for probability modeling. The GMM consists of multiple Gaussian components, and each component has its own weight, mean vector, and covariance matrix. These parameters are recursively updated through a long short-term memory network (LSTM), and the LSTM can capture long-term dependencies in time series data. Through the update of the LSTM, the GMM can learn the probability distribution of the spatio-temporal features. For example, suppose the GMM has 16 Gaussian components, then each component has a weight, a 256-dimensional mean vector, and a 256x256 covariance matrix. The LSTM updates these parameters according to the current spatio-temporal feature sequence to obtain the probability distribution of the spatio-temporal features.

[0076] The probability distribution of spatio-temporal features and acoustic features are input into a conditional variational autoencoder (CVAE). The CVAE includes an encoder and a decoder. The encoder encodes the probability distribution of spatio-temporal features and acoustic features into high-level feature representations and outputs the parameters of the posterior distribution. The decoder reconstructs the probability distribution of spatio-temporal features based on the sampled latent variables and acoustic features. The CVAE is trained by maximizing the evidence lower bound (ELBO). The ELBO consists of a reconstruction term and a KL divergence term. The reconstruction term measures the similarity between the reconstructed probability distribution of spatio-temporal features and the original distribution, and the KL divergence term measures the difference between the posterior distribution and the prior distribution. By calculating the gradient norms of the ELBO with respect to the probability distribution of spatio-temporal features and acoustic features, the modal importance scores can be obtained. Dynamically adjusting the weight of the KL divergence term according to the modal importance scores can make the model pay more attention to important modal information. The output of the trained CVAE serves as the joint probability features of actions and sounds.

[0077] Through multimodal deep fusion and attention mechanism, the present invention can effectively fuse information of different modalities, improve the accuracy of action recognition. Especially in complex scenarios, through dynamic weight allocation and probability modeling, it can effectively process noise and missing data, enhance the robustness of the model. Through the attention mechanism and probability modeling, it can better understand the contributions of different modalities to action recognition and improve the interpretability of the model.

[0078] In an alternative embodiment,

[0079] The steps of obtaining multi-dimensional combined features by processing the joint probability features through the temporal knowledge graph of the graph attention network and inputting the multi-dimensional combined features into a two-stream autoencoder network for feature mapping of the source scene and obtaining the target scene features using adversarial training include:

[0080] Input the joint probability features into the temporal knowledge graph, calculate the correlation weights between temporal nodes through the attention mechanism, where the temporal nodes correspond to feature vectors at different times; process the weighted temporal knowledge graph through the multi-head attention mechanism of the multi-layer graph attention network, and obtain multi-dimensional combined features through multi-layer stacking;

[0081] Input the multi-dimensional combined features into the two-stream autoencoder network, where the source scene encoder branch consists of multi-layer perceptrons, encodes the input features into source scene latent features through layer-by-layer non-linear transformation, and the target scene adapter branch introduces residual connections in the encoder, adds the source scene latent features and the encoder output features of the target scene adapter branch to obtain the target scene latent features, and reconstructs the target scene features through the decoder network;

[0082] Input the target scene features into a discriminator network with a block discriminant structure, perform local discrimination on the features through multiple layers of convolution, calculate the Wasserstein distance between the source scene latent features and the target scene latent features as the adversarial loss, and introduce a gradient penalty term;

[0083] Calculate the maximum mean discrepancy between the source scene and the target scene latent features as the distribution alignment constraint, and at the same time introduce the reconstruction constraint, the cycle consistency constraint, and the contrastive learning loss;

[0084] Alternately optimize the adversarial loss of the discriminator network and the joint loss of the target scene adapter through exponential moving average. The joint loss includes the adversarial loss, the distribution alignment loss, the cycle consistency loss, and the contrastive loss.

[0085] Exemplarily, construct a temporal knowledge graph. Arrange the input joint probability features in chronological order, and the feature vector at each moment forms a node in the graph. Then, calculate the temporal correlation between the nodes. For example, the cosine similarity of the two node feature vectors can be used to measure their correlation, and the calculation result is used as the weight of the edge connecting the two nodes. In this way, a weighted temporal knowledge graph is constructed. Assume that the joint probability feature dimension is 128 and the time step is 10, then there are 10 nodes, and the feature dimension of each node is 128. The weights of the connecting edges between the nodes are, for example, 0.8, 0.5, 0.9, etc.

[0086] Input the constructed temporal knowledge graph into a multi-layer graph attention network. This network uses the multi-head attention mechanism to transform and aggregate the node features. Each attention head focuses on different aspects of the node, and the results of multiple attention heads are concatenated to obtain a more comprehensive node representation. Through multi-layer stacking, multi-layer transmission of the features can be achieved, and finally multi-dimensional combined features are obtained. For example, after three layers of graph attention network, the finally output feature dimension is 512.

[0087] Input the obtained 512-dimensional multi-dimensional combined features into a two-stream autoencoder network. This network includes a source scene encoder branch and a target scene adapter branch. The source scene encoder branch is composed of multi-layer perceptrons, and the input multi-dimensional combined features are encoded into low-dimensional source scene latent features, such as 256 dimensions, through layer-by-layer non-linear transformation. The target scene adapter branch also includes an encoder, whose structure is similar to that of the source scene encoder, but a residual connection is introduced, and the source scene latent features are added to the output of the encoder of the target scene adapter branch to obtain the target scene latent features. Finally, the target scene latent features are reconstructed through the decoder network to obtain the target scene features, and the dimension is the same as the dimension of the input multi-dimensional combined features, that is, 512 dimensions.

[0088] To train the target scene adapter, an adversarial training mechanism is introduced. The reconstructed target scene features are input into the discriminator network, which adopts a block discriminant structure. The features are locally discriminated through multiple layers of convolution, and the Wasserstein distance between the source scene latent features and the target scene latent features is calculated as the adversarial loss. To stabilize the training process, a gradient penalty term is introduced in the adversarial training. At the same time, the maximum mean discrepancy metric between the source scene latent features and the target scene latent features is calculated as the distribution alignment constraint, for example, the Gaussian kernel function is used to calculate the feature distance. In addition, a reconstruction constraint is imposed on the source scene latent features, and a cycle consistency constraint is imposed on the target scene latent features, that is, the target scene latent features are input into the target scene adapter again, and the output is required to be as close as possible to the original input. To enhance the discriminability of the features, a contrastive learning loss is also introduced.

[0089] The network is trained by jointly optimizing the loss function. Specifically, the reconstruction loss and the contrastive loss of the source scene encoder branch are optimized, and the adversarial loss of the discriminator network and the joint loss of the target scene adapter are alternately optimized. The joint loss of the target scene adapter includes the adversarial loss, the distribution alignment loss, the cycle consistency loss, and the contrastive loss. To stabilize the training process, the exponential moving average is used to update the parameters of the target scene adapter branch and the discriminator network.

[0090] The present invention effectively captures the temporal dependence relationship of features through the temporal knowledge graph, and uses the dual-stream autoencoder and the adversarial training mechanism to achieve the effective adaptation of the source scene features to the target scene features, thereby improving the cross-scene and cross-domain feature transfer learning ability. Through the multi-layer graph attention network and the multi-head attention mechanism, more comprehensive and robust feature representations can be extracted, effectively reducing the influence of noise and irrelevant information, and improving the stability and generalization ability of the feature representation. Through various mechanisms such as adversarial training, distribution alignment constraint, cycle consistency constraint, and contrastive learning, the distributions of the source scene features and the target scene features are made as close as possible, so as to better achieve cross-scene feature adaptation and improve the transfer learning performance of the model.

[0091] In an alternative embodiment,

[0092] The steps of inputting the source scene features and the target scene features into the memory-augmented neural network and outputting the scene adaptive features by combining the reinforcement learning method include:

[0093] Input the source scene features and the target scene features into the hierarchical memory writing mechanism of the memory-augmented neural network for feature normalization processing, obtain the fused features through the multi-head attention mechanism, determine the target memory slot based on the locality-sensitive hashing, and write the fused features into the memory module through the gated writing mechanism;

[0094] Input the memory module into the reinforcement learning framework, which includes an Actor-Critic network structure and a multi-step memory reasoning mechanism. The Actor network generates a memory operation policy based on the current memory content, the fusion feature, and the historical operation sequence. The Critic network uses a dual value evaluation network structure to evaluate the state-action value function, constructs a reward function based on feature mutual information, distribution distance, and memory timeliness, updates the policy using policy gradients, and performs multi-step memory reasoning through temporal difference.

[0095] Construct a multi-scale query vector for the fusion feature. The multi-scale query vector includes a spatial scale vector, a temporal scale vector, and a channel scale vector. Retrieve the memory feature with the maximum cosine similarity to the query vector from the memory module, integrate it with the fusion feature through an adaptive fusion network, and obtain the scene adaptive feature through a residual connection.

[0096] Optimize the memory module based on the weighted combination of memory duration and feature mutual information, including priority update, memory compression, and dynamic adjustment of memory capacity.

[0097] Exemplarily, perform feature preprocessing and memory writing. The source scene feature and the target scene feature are respectively input into the memory-augmented neural network. To eliminate the influence of different feature dimensions and dimensions, perform feature normalization on the source scene feature and the target scene feature. For example, scale each feature dimension to zero mean and unit variance. Then, use the multi-head attention mechanism to interact the normalized source scene feature and target scene feature to obtain the fusion feature. The multi-head attention mechanism allows the network to focus on the feature interactions in different subspaces, thereby capturing richer feature representations. For example, the source scene feature is an image feature vector with a dimension of 1024, and the target scene feature is a text feature vector with a dimension of 512. Through the multi-head attention mechanism, the two can be fused into a feature vector with a dimension of 2048, containing the interaction information of the image and the text. Next, use the locality-sensitive hashing method to calculate multiple hash values for the fusion feature. For example, calculate 8 hash values. Determine the target memory slot according to these hash values. For example, write the fusion feature into the 8 memory slots corresponding to the hash values. Finally, write the fusion feature into the memory module through the gated writing mechanism. The gated writing mechanism controls which information is written into the memory to avoid the memory being overwritten by irrelevant information.

[0098] Perform reinforcement learning training. Input the memory module into the reinforcement learning framework, which includes an Actor-Critic network structure and a multi-step memory inference mechanism. The Actor network generates a memory operation policy based on the current memory content, fused features, and historical operation sequences. For example, it decides which information to read from or write to the memory. The Critic network uses a dual value evaluation network structure to evaluate the state-action value function, where the feature value network evaluates the value of the current state, and the memory value network evaluates the value of the memory content. For example, the feature value network judges the favorability of the scene based on the current scene features, and the memory value network judges the potential benefits of the current action based on the historical experience in the memory. To guide the reinforcement learning training, a comprehensive reward function is constructed, which is based on the mutual information between features and memory, the distribution distance between source scene features and target scene features, and the memory timeliness metric. For example, if the information in the memory is highly correlated with the current scene features, a high reward is given; if the distribution of source scene and target scene features is very different, a high reward is given to encourage the network to learn cross-scene adaptability; if the information in the memory was recently written, a high reward is given. Use the policy gradient method to update the memory operation policy. For example, use the REINFORCE algorithm to update the parameters of the Actor network. Perform multi-step memory inference through the temporal difference method. For example, use the TD(λ) algorithm to estimate the long-term reward. And perform memory tracking based on the product of the historical state importance cumulative sum and the decay factor. For example, assign different weights to past states and focus on the states that have an important impact on the current decision.

[0099] Perform feature retrieval and fusion. Construct multi-scale query vectors for the fused features, including spatial scale vectors, temporal scale vectors, and channel scale vectors. For example, for image features, the spatial scale vector can represent different regions of the image, the temporal scale vector can represent different time frames of the image, and the channel scale vector can represent different channels of the image. Retrieve memory features with the maximum cosine similarity to the query vectors from the memory module based on the multi-scale query vectors. For example, calculate the cosine similarity between each query vector and all feature vectors in the memory, and select the feature vector with the highest similarity as the retrieval result. Integrate the retrieved memory features with the fused features through an adaptive fusion network. Use a fully connected network to concatenate the fused features and the retrieved memory features and then perform a non-linear transformation. Finally, obtain the scene adaptive features through a residual connection, adding the fused features and the output of the adaptive fusion network to get the final scene adaptive features.

[0100] Memory optimization is performed. The memory module is optimized based on the weighted combination of memory duration and feature mutual information, including priority update, memory compression, and dynamic adjustment of memory capacity. For example, for memories with long duration and high mutual information with the current feature, their priorities are increased; for memories with short duration and low mutual information with the current feature, their priorities are decreased, and they are compressed or deleted to free up memory space. The memory capacity can be dynamically adjusted to automatically adjust the size of the memory module according to task requirements and computing resource limitations.

[0101] By combining the information of the source scene and the target scene and using the memory mechanism for knowledge accumulation and transfer, the present invention can extract more comprehensive and discriminative scene-adaptive features, effectively improve the feature representation ability. Through reinforcement learning training and memory optimization, it can adapt to the changes in different scenes, improve the generalization ability of the model, and enable it to maintain good performance in new scenes. Through techniques such as locality-sensitive hashing and multi-scale queries, it can quickly retrieve and access relevant information in memory, improve computing efficiency, and reduce computing costs.

[0102] In an alternative embodiment,

[0103] The Actor network generates a memory operation policy based on the current memory content, the fusion feature, and the historical operation sequence. The Critic network uses a dual-value evaluation network structure to evaluate the state-action value function. The dual-value evaluation network includes a feature value network and a memory value network. The steps of constructing a comprehensive reward function based on the mutual information between features and memory, the distribution distance between source scene features and target scene features, and the memory timeliness metric, updating the memory operation policy using the policy gradient method, performing multi-step memory reasoning through the temporal difference method, and memory tracking based on the product of the historical state importance accumulation and the decay factor include:

[0104] The Actor network encodes the current memory content through a multi-layer perceptron to obtain a memory representation, performs feature transformation on the fusion feature through a linear transformation to obtain a fusion feature representation, encodes the historical operation sequence through a gated recurrent unit network to obtain a historical representation, concatenates the memory representation, the fusion feature representation, and the historical representation, and then obtains a memory operation policy distribution through a fully connected layer; constructs a Critic network using a dual-value evaluation network, including a feature value network based on feature mutual information and distribution distance, and a memory value network based on storage duration and usage frequency, and obtains the state-action value function by combining the outputs of the feature value network and the memory value network through a learnable balance factor;

[0105] Construct a comprehensive reward function based on multi-factor coupling. The comprehensive reward function includes: an information mutual measure term obtained by calculating the joint distribution and marginal distribution of features and memories through kernel density estimation, a distribution distance term of source scene features and target scene features in the reproducing kernel Hilbert space calculated using the maximum mean discrepancy, and a memory timeliness metric term modeled based on the exponential decay function; design a grid search-based cross-validation method to determine the weight coefficients of the comprehensive reward function, perform five-fold cross-validation within a preset weight search space, use a weighted combination of anomaly detection accuracy, F1 score, and area under the curve as the validation metric, and introduce a dynamic weight adjustment mechanism to adaptively update the initial weights according to the change rates of mutual information, distribution distance, and memory timeliness.

[0106] Adopt the n-step temporal difference method to construct the cumulative reward, combine the rewards from the current state to the next n states with the decay values of the state-action value function, construct an optimization objective based on the difference between the cumulative reward and the state-action value function, and iteratively update the Actor network parameters through the policy gradient method; perform multi-step memory reasoning based on the temporal difference method, and iteratively update the memory operation policy by calculating the temporal difference error between the current state and the target state; introduce a memory tracking mechanism based on importance sampling, calculate the importance weights of each state in the historical state sequence, where the importance weight is the ratio of the current policy probability to the behavior policy probability, use the cumulative product of the importance weight and the decay factor as the state importance metric, and track the memory operation based on the state importance.

[0107] Exemplarily, a memory reasoning method based on the Actor-Critic framework and the multi-factor coupling reward function is used to improve the decision-making efficiency and accuracy in complex scenarios. Its core lies in making decisions by combining feature information and memory information, and optimizing the memory operation policy through multi-step reasoning and the memory tracking mechanism.

[0108] Construct an Actor network and a Critic network. The Actor network is responsible for generating the memory operation policy, and it receives the current memory content, fused features, and historical operation sequence as inputs. Specifically, the Actor network uses a multi-layer perceptron to encode the current memory content to obtain a memory representation; uses a linear transformation to perform feature transformation on the fused features to obtain a fused feature representation; uses a gated recurrent unit network to encode the historical operation sequence to obtain a historical representation. Then, these three representations are concatenated together and passed through a fully connected layer to finally output the probability distribution of the memory operation policy.

[0109] The Critic network is responsible for evaluating the state-action value function. It adopts a dual-value evaluation network structure, including a feature value network and a memory value network. The feature value network is evaluated based on the mutual information between features and memories, as well as the distribution distance between source-scene features and target-scene features. For example, the kernel density estimation method can be used to calculate the joint distribution and marginal distribution of features and memories, thereby obtaining the mutual information. The maximum mean discrepancy can be used to calculate the distribution distance between source-scene features and target-scene features in the reproducing kernel Hilbert space. The memory value network is evaluated based on the storage duration and usage frequency of memories. For example, the memory value can be calculated according to the length of time the memory is stored and the number of times it is used. Finally, the outputs of the feature value network and the memory value network are combined through a learnable balance factor to obtain the state-action value function.

[0110] Construct a comprehensive reward function. This function consists of three parts: the mutual information measurement term between features and memories, the distribution distance term between source-scene features and target-scene features, and the memory timeliness measurement term. The mutual information measurement term is used to measure the correlation between features and memories. The distribution distance term is used to measure the difference between the source scene and the target scene. The memory timeliness measurement term is used to measure the effectiveness of memories. For example, an exponential decay function can be used to model the memory timeliness. To determine the weight coefficients of these three terms in the comprehensive reward function, the method of grid search and five-fold cross-validation is adopted. In the preset weight search space, five-fold cross-validation is carried out, and the weighted combination of anomaly detection accuracy, F1 score, and area under the curve is used as the validation index to select the best weight coefficients. In addition, a dynamic weight adjustment mechanism is introduced to adaptively update the initial weights according to the change rates of mutual information, distribution distance, and memory timeliness.

[0111] Use the n-step temporal difference method to construct the cumulative reward, and take the difference between the cumulative reward and the state-action value function as the optimization objective, and iteratively update the Actor network parameters through the policy gradient method. For example, the weighted sum of the rewards from the current state to the next n states and the discounted values of the state-action value function can be calculated as the cumulative reward.

[0112] Based on the temporal difference method, perform multi-step memory reasoning, and iteratively update the memory operation policy by calculating the temporal difference error between the current state and the target state.

[0113] Introduce a memory tracking mechanism based on importance sampling. Calculate the importance weight of each state in the historical state sequence, which is the ratio of the current policy probability to the behavior policy probability. Take the cumulative product of the importance weight and the decay factor as the state importance measure, and track the memory operation based on the state importance.

[0114] As Figure 2As shown in the adaptability comparison chart after scene transformation, this chart shows the accuracy recovery trends of three different methods (the method in this paper, the basic memory network, and the spatio-temporal convolutional network STCN) as the number of iterations increases after scene transformation. The horizontal axis represents the number of iterations (0 - 800 times), and the vertical axis represents the detection accuracy (50% - 100%). Different line types and point markers are used in the figure to distinguish the three methods: The method in this paper (solid line + circular marker): The initial accuracy is about 95%, drops to 89% after scene transformation, and then quickly recovers, returning to the original level after about 500 iterations. The basic memory network (long dashed line + triangular marker): The initial accuracy is about 90%, drops significantly to 71% after scene transformation, and has a slow recovery speed, only recovering to 87% after 800 iterations. The spatio-temporal convolutional network STCN (short dashed line + square marker): The initial accuracy is about 85%, drops significantly to 57% after scene transformation, and has a very slow recovery speed, only recovering to 78% after 800 iterations.

[0115] The chart clearly shows three significant advantages of the method in this paper after scene transformation: 1) The performance drops slightly, only dropping by about 6 percentage points; 2) The recovery speed is fast, reaching a stable state after about 300 - 400 iterations; 3) The recovery degree is high, being able to fully recover to the original performance level. In contrast, the basic memory network drops by about 19 percentage points, and the spatio-temporal convolutional network drops by about 28 percentage points, and neither can fully recover within 800 iterations. The present invention can still maintain stable detection performance in scenarios such as environmental changes, furniture layout adjustments, and new user additions, greatly reducing the need for system retraining and providing a more stable and reliable solution for practical applications. Reducing adaptation time and computational resource requirements: The fast adaptability (only an average of 312 iterations required) means that the system can adapt to new scenarios with fewer computational resources and time, which is particularly important for edge devices with limited computational resources, while reducing the system unavailable time. Reducing maintenance costs and user intervention: Traditional methods often require manual intervention or reconfiguration after scene changes, while the adaptive ability of the present invention can significantly reduce the need for maintenance costs and professional personnel intervention, making the system more suitable for long-term deployment in a home environment. Improving the continuity of anomaly detection: During scene transformation, the present invention can still maintain a high accuracy (the lowest point is 89%), ensuring the continuity and reliability of anomaly detection and avoiding a large number of missed reports or false alarms of traditional methods after environmental changes.

[0116] As Figure 3 shown in the Monte Carlo simulation result chart, Figure 3Shows the detection accuracy distribution of three different algorithms through 1000 random scenario tests using the Monte Carlo method. The horizontal axis represents the random scenario sample index (0 - 100), and the vertical axis represents the detection accuracy (60% - 100%). In the figure, the three methods are distinguished by different marker shapes rather than color depths: The method of this paper (circular markers): The detection accuracy is concentrated in the range of 88.5% - 95.3%, showing a highly concentrated distribution characteristic, and the standard deviation is only 1.8%. The basic memory network (triangular markers): The detection accuracy is distributed in the range of 74.8% - 89.7%, with a relatively dispersed distribution, and the standard deviation is 3.9%. The spatio-temporal convolutional network STCN (square markers): The detection accuracy is distributed in the range of 62.5% - 85.4%, with a highly dispersed distribution, and the standard deviation is as high as 5.7%. The performance stability of the three methods under various random perturbations is intuitively shown through the scatter distributions of different shapes. The scatter points of the method of this paper (circular markers) are highly concentrated in the high-accuracy region, indicating its strong robustness to various random scenario changes. In contrast, the scatter points of the basic memory network (triangular markers) and the spatio-temporal convolutional network (square markers) are more dispersed and the overall accuracy is lower, indicating that these two methods are more sensitive to scenario changes. In addition, the lowest accuracy (88.5%) of the method of this paper is still higher than the average accuracy of the other two methods, demonstrating its excellent stability. The system reliability of the present invention is significantly improved, reducing the risks of false alarms and missed detections, reducing the need for parameter tuning, and improving the long-term operation performance.

[0117] In an alternative embodiment,

[0118] The steps of inputting the scene adaptive feature into a multi-task learning network, extracting comprehensive features through a shared representation layer, outputting an anomaly prediction result through a task attention layer, and inputting the anomaly prediction result into a risk propagation neural network to analyze the coupling relationship of risk factors and output the risk level include:

[0119] Based on the scene adaptive feature, local to global features are extracted through the multi-scale convolutional kernels of the shared representation layer of the multi-task learning network, and the comprehensive features are obtained through fusion by a feature pyramid network; the attention scores are calculated using a multi-head collaborative attention mechanism, and after integrating the multi-head outputs, they are fused with the comprehensive features through a residual gating mechanism to obtain the anomaly prediction result; the anomaly prediction result is input into the risk propagation neural network, a risk factor relationship graph is constructed, and the risk factor nodes and the association relationships between factors are processed through a node feature network and an edge feature network respectively to obtain the initial node features and the edge features of the association relationships between risk factors;

[0120] Adopt the message passing mechanism of gated graph convolution, where the message generation sub-module combines the source node features, target node features and edge features through a gated unit, and the message aggregation sub-module weights and aggregates the neighborhood message features based on the attention coefficients between nodes;

[0121] Iteratively update the node features of the risk factor relationship graph through multiple rounds of message passing, and add the risk factor node features after a preset number of rounds of message passing to the initial risk factor node features after residual mapping to obtain the risk factor node states; calculate the risk propagation intensity between risk factor nodes based on the feature similarity between risk factor node states and the corresponding edge features, and integrate it through hierarchical graph pooling to obtain the risk level.

[0122] Exemplarily, perform data preprocessing. Collect historical data containing various abnormal events and risk factors, and perform preprocessing operations such as data cleaning and normalization on the data. Then, construct scene-adaptive features, select appropriate features according to the specific scene, and at the same time, considering the particularity of the scene, introduce features that can reflect scene changes.

[0123] Use a multi-task learning network to extract comprehensive features. The shared representation layer of this network uses convolutional kernels of different sizes to extract local and global features, and performs feature fusion through a feature pyramid network.

[0124] Construct a task attention layer. This layer contains a multi-head collaborative attention mechanism. Each attention head independently calculates the attention score, and integrates the outputs of multiple attention heads. Then, use a residual gating mechanism to fuse the integrated features with the comprehensive features to obtain the anomaly prediction result.

[0125] Input the anomaly prediction result into the risk propagation neural network. First, construct a risk factor relationship graph to represent the association relationship between risk factors. Then, use a node feature network and an edge feature network to process risk factor nodes and the association relationship between factors respectively to obtain the initial node features and edge features. For example, a neural network can be used to extract the features of each risk factor and the features of the association relationship between risk factors.

[0126] Design a message passing mechanism based on gated graph convolution. The message generation sub-module combines the source node features, target node features, and edge features to generate message features. The message aggregation sub-module performs weighted summation on the message features of neighboring nodes based on the attention coefficient. For example, the feature of a risk factor is affected by the risk factors adjacent to it, and this influence is simulated through the message passing mechanism.

[0127] Perform multiple rounds of message passing iteration on the risk factor relationship graph to update the risk factor node features. For example, after multiple rounds of iteration, the features of each risk factor will be updated, thus reflecting the propagation process of risks in the network.

[0128] Sum the iterated risk factor node features with the initial features to obtain the risk factor node state. For example, sum the risk factor node features after multiple rounds of message passing with the initial features to obtain the final risk factor node state.

[0129] Calculate the risk propagation intensity between risk factor nodes based on the risk factor node state and edge features. For example, calculate the risk propagation intensity according to the similarity of the risk factor node states and their association relationships.

[0130] Adopt a hierarchical graph pooling operation to integrate the node-level risk state and risk propagation intensity to obtain the risk level. For example, integrate the risk factor node state and risk propagation intensity to finally obtain a numerical value representing the risk level.

[0131] By combining scene adaptive features and multi-task learning, the present invention can more accurately identify abnormal events in different scenarios, thereby improving the accuracy of anomaly detection; by considering the coupling relationship between risk factors and the risk propagation process, it can more comprehensively evaluate the risk level and avoid missing potential risk factors; by grading risks, corresponding measures can be taken more targeted, thereby improving the efficiency of risk management.

[0132] In an alternative embodiment,

[0133] The steps of inputting the risk level into a recurrent neural network, performing feature memory through a gating mechanism in combination with historical warning feedback information, and outputting a warning strategy include:

[0134] Encode the risk level through a bidirectional long short-term memory network to obtain a fused time series representation; input the historical warning feedback information into a feature transformation network to obtain a feedback feature representation; calculate attention weights at each layer of the time granularity pyramid based on the fused time series representation and the feedback feature representation, and weighted integrate through a soft attention mechanism to obtain a multi-granularity memory feature;

[0135] Design a forgetting gate and an update gate to dynamically update the multi-granularity memory feature, where the forgetting gate controls the retention degree of historical information based on the fused time series representation and the multi-granularity memory feature, and the update gate controls the introduction ratio of new knowledge to obtain an updated memory feature; input the fused time series representation and the updated memory feature into a gated recurrent unit for adaptive fusion to obtain a warning state representation;

[0136] Construct a multi-objective early warning optimization function. The multi-objective early warning optimization function comprehensively considers the timeliness of early warning, the accuracy of early warning, and the recognition rate of abnormal behaviors. Input the early warning status representation into a deep Q-learning network, construct a state-action value function based on scenario adaptive features and early warning response behaviors. The state of the state-action value function includes scenario adaptive features and environmental parameters, and the actions of the state-action value function include early warning level adjustment and intervention measure recommendation. Map the state-action value function to obtain an early warning strategy, and iteratively optimize the early warning strategy through the temporal difference algorithm to obtain the optimal early warning strategy.

[0137] Exemplarily, encode the input risk level sequence. The risk level sequence can be represented as a series of numerical values, for example: [1, 2, 3, 4, 5], where the larger the numerical value, the higher the risk level. Input this sequence into a bidirectional long short-term memory network (BiLSTM). The BiLSTM network contains two LSTM layers, a forward layer and a backward layer. The forward LSTM layer processes the risk level sequence in chronological order, extracts the temporal dependencies in the sequence, and obtains a series of forward hidden state vectors. The backward LSTM layer processes the risk level sequence in reverse order, extracts the reverse temporal dependencies, and obtains a series of backward hidden state vectors. Concatenate the forward hidden state vector and the backward hidden state vector at the corresponding time to obtain a representation that fuses bidirectional temporal information, denoted as the fused temporal representation.

[0138] Process the historical early warning feedback information. The historical early warning feedback information includes the early warning time, early warning level, response measures, and effect evaluation information. Input this information into a feature transformation network, which can be a multi-layer perceptron or a convolutional neural network, etc. The feature transformation network converts this information into a feature vector with a fixed dimension, denoted as the feedback feature representation. For example, after passing through the feature transformation network, the above historical early warning feedback information may be converted into a 128-dimensional vector.

[0139] Construct a time granularity pyramid. The time granularity pyramid can be understood as observing the time series at different scales. For example, the risk level sequence can be analyzed at the granularities of days, weeks, and months. At each time granularity layer, calculate the attention weights based on the fused temporal representation and the feedback feature representation. For example, at the "day" granularity layer, calculate the correlation between the risk level of each day and the historical early warning feedback information to obtain the attention weight for each day. The attention weight reflects the importance of different time points. Through the soft attention mechanism, weight and integrate the attention features at different time granularities to obtain the multi-granularity memory feature. For example, weight and sum the attention features at the three granularities of "day", "week", and "month" to obtain the final multi-granularity memory feature.

[0140] To enable the model to dynamically update its memory, a forgetting gate and an update gate are designed to dynamically update the multi-granularity memory features. The forgetting gate controls the degree of retention of historical information based on the fused temporal representation and multi-granularity memory features. For example, if the current risk level is very high, the forgetting gate may reduce the weight of historical information. The update gate controls the proportion of new knowledge introduced. For example, if the current warning feedback information is very important, the update gate may increase the weight of new information. Through the forgetting gate and the update gate, the updated memory features are obtained.

[0141] The fused temporal representation and the updated memory features are input into a gated recurrent unit (GRU) for adaptive fusion to obtain the warning state representation. The GRU can effectively fuse information from different sources and capture long-term dependencies in the sequence.

[0142] A multi-objective warning optimization function is constructed. This function comprehensively considers warning timeliness, warning accuracy, and the abnormal behavior recognition rate. The warning state representation is input into a deep Q-learning network. The deep Q-learning network constructs a state-action value function based on scenario-adaptive features and warning response behaviors. The states of the state-action value function include scenario-adaptive features and environmental parameters, such as the current risk level, the number of historical warnings, etc. The actions include warning level adjustment and intervention measure recommendations, such as raising the warning level from level one to level two, or recommending a certain emergency measure. The deep Q-learning network maps the state-action value function to obtain a warning strategy. The warning strategy is iteratively optimized through the temporal difference algorithm to obtain the optimal warning strategy. For example, if a certain warning strategy can give an early warning and effectively reduce the risk, the value of this strategy will be improved.

[0143] Traditional time series analysis mainly uses unidirectional LSTM or simple RNN to process time series data, and it is unable to comprehensively capture the two-way evolution law of risk sequences. Considering that risk sequences contain both historical cumulative effects and future development trends, and risks exhibit different evolution characteristics at different time scales, the present invention proposes an early warning method based on bidirectional long short-term memory network and multi-granularity time series analysis. First, the BiLSTM structure is used to extract time series dependencies through forward and backward LSTM layers respectively to achieve two-way analysis of risk sequences. Secondly, a feature transformation network is designed to convert historical information such as early warning time, level, measures, and effects into a unified feature representation. Innovatively, a time granularity pyramid is constructed to analyze the risk evolution characteristics at different time scales such as days, weeks, and months, and multi-scale features are adaptively fused through a soft attention mechanism. The present invention designs a dynamic update mechanism based on the forget gate and update gate, and adaptively controls the retention of historical information and the introduction of new knowledge by evaluating the importance of the current risk level and historical information. This mechanism enables the system to quickly respond to new risk situations while retaining important historical experiences. The present invention uses the deep Q-learning method to construct a state-action value function, and transforms the early warning problem into a multi-objective optimization problem. By considering scenario characteristics and environmental parameters, the system can adaptively adjust the early warning level and recommend intervention measures, and continuously optimize the early warning strategy through the temporal difference algorithm.

[0144] In the second aspect of the embodiments of the present invention, a home abnormal state signal detection system based on multi-modal sensing is provided. The system includes:

[0145] A first unit for inputting the collected human motion data, acoustic features, and environmental parameters into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatio-temporal attention mechanism, extracts a spatio-temporal feature sequence through a multi-scale convolutional neural network, and combines it with the spectral analysis result of acoustic features to form a joint probability feature;

[0146] A second unit for processing the joint probability feature through the temporal knowledge graph of the graph attention network to obtain a multi-dimensional combined feature, inputting the multi-dimensional combined feature into a two-stream autoencoder network, performing feature mapping on the source scene and obtaining target scene features through adversarial training; inputting the source scene features and target scene features into a memory-augmented neural network, and outputting scene adaptive features by combining a reinforcement learning method;

[0147] A third unit for inputting the scene adaptive feature into a multi-task learning network, extracting a comprehensive feature through a shared representation layer, outputting an abnormal prediction result through a task attention layer, inputting the abnormal prediction result into a risk propagation neural network, analyzing the coupling relationship of risk factors through a message passing mechanism and outputting a risk level; inputting the risk level into a recurrent neural network, and performing feature memory through a gating mechanism by combining historical early warning feedback information and outputting an early warning strategy.

[0148] In the third aspect of the embodiments of the present invention,

[0149] a kind of electronic device is provided, including:

[0150] a processor;

[0151] a memory for storing instructions executable by the processor;

[0152] wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0153] In the fourth aspect of the embodiments of the present invention,

[0154] a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0155] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for executing various aspects of the present invention are uploaded.

[0156] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal state signals at home based on multimodal sensing, characterized in that, Including: Input the collected human motion data, acoustic features, and environmental parameters into a deep fusion network. The deep fusion network dynamically assigns weights to different features through a spatio-temporal attention mechanism, extracts spatio-temporal feature sequences through a multi-scale convolutional neural network, and combines them with the spectral analysis results of acoustic features to form joint probability features; Process the joint probability features through the temporal knowledge graph of the graph attention network to obtain multi-dimensional combined features. Input the multi-dimensional combined features into a two-stream autoencoder network, perform feature mapping on the source scene, and use adversarial training to obtain target scene features; Input the source scene features and target scene features into a memory-augmented neural network, and combine reinforcement learning methods to output scene adaptive features; Input the scene adaptive features into a multi-task learning network, extract comprehensive features through a shared representation layer, output anomaly prediction results through a task attention layer, input the anomaly prediction results into a risk propagation neural network, analyze the coupling relationship of risk factors through a message passing mechanism and output a risk level; Input the risk level into a recurrent neural network, and combine historical warning feedback information to perform feature memory through a gating mechanism and output a warning strategy.

2. The method according to claim 1, characterized in that, The steps of inputting the collected human motion data, acoustic features, and environmental parameters into a deep fusion network to form joint probability features include: Perform filtering, transformation, and normalization processing on human motion data, acoustic signals, and environmental parameters to obtain normalized features; Pass the normalized features through the time attention layer and the modality attention layer of a two-layer attention network in sequence to obtain weighted multi-modal features; Input the multi-modal features into a multi-branch convolutional neural network, and extract spatio-temporal feature sequences through depthwise separable convolution and residual connections; Input the spatio-temporal feature sequences into a Gaussian mixture model. The Gaussian mixture model includes multiple Gaussian components, each Gaussian component has a corresponding mixing weight, mean vector, and covariance matrix. The mean vector and covariance matrix are both functions of latent variables. Recursively update the latent variables through a long short-term memory network to obtain the spatio-temporal feature probability distribution; Input the spatio-temporal feature probability distribution and acoustic features into a conditional variational autoencoder. The conditional variational autoencoder includes a two-branch encoder network. The conditional variational autoencoder obtains high-level feature representations through the two-branch encoder network, and outputs posterior distribution parameters through a mean network and a variance network; Perform reparameterization sampling on the posterior distribution parameters and reconstruct them through a decoder network, and calculate the evidence lower bound of the reconstruction term and the KL divergence; Calculate the gradient norm of the evidence lower bound with respect to the spatio-temporal feature probability distribution and acoustic features to obtain the modality importance score, dynamically adjust the KL divergence weight, and maximize the evidence lower bound to train the conditional variational autoencoder to obtain the joint probability features of actions and sounds.

3. The method according to claim 1, characterized in that, The steps of processing the joint probability features through the temporal knowledge graph of the graph attention network to obtain multi-dimensional combined features, inputting the multi-dimensional combined features into a two-stream autoencoder network, performing feature mapping on the source scene, and using adversarial training to obtain target scene features include: The joint probability features are input into the time series knowledge graph, and the correlation weights between the time series nodes are calculated through the attention mechanism. The time series nodes correspond to feature vectors at different times. The weighted time series knowledge graph is processed by the multi-head attention mechanism of the multi-layer graph attention network, and multi-dimensional combined features are obtained by multi-layer stacking. The multi-dimensional combined features are input into a two-stream autoencoder network, wherein the source scene encoder branch is composed of a multi-layer perceptron, and the input features are encoded into source scene latent features through layer-by-layer nonlinear transformation, and the target scene adapter branch introduces a residual connection in the encoder, and the source scene latent features and the encoder output features of the target scene adapter branch are added to obtain the target scene latent features, and the target scene features are reconstructed through a decoder network; Inputting the target scene features into a discriminator network with a block discriminant structure, locally discriminating the features through multi-layer convolution, calculating the Wasserstein distance between the source scene potential features and the target scene potential features as adversarial loss and introducing a gradient penalty term; The maximum average difference between the latent features of the source scene and the target scene is calculated as a distribution alignment constraint, and reconstruction constraints, cycle consistency constraints and contrastive learning losses are introduced at the same time; The adversarial loss of the discriminator network and the joint loss of the target scene adapter are optimized alternately by exponential sliding average, and the joint loss includes adversarial loss, distribution alignment loss, cycle consistency loss and contrastive loss.

4. The method according to claim 1, wherein The steps of inputting source scene features and target scene features into a memory-enhanced neural network and outputting scene adaptive features by combining a reinforcement learning method include: The source scene features and the target scene features are input into the hierarchical memory writing mechanism of the memory-enhanced neural network for feature normalization. The fused features are obtained through interaction through the multi-head attention mechanism. The target memory slot is determined based on the local sensitive hashing, and the fused features are written into the memory module through the gated writing mechanism. The memory module is input into a reinforcement learning framework, wherein the reinforcement learning framework includes an Actor-Critic network structure and a multi-step memory reasoning mechanism, wherein the Actor network generates a memory operation strategy based on the current memory content, the fusion features and the historical operation sequence, and the Critic network uses a dual value evaluation network structure to evaluate the state action value function, constructs a reward function based on feature mutual information, distribution distance and memory timeliness, and uses a policy gradient update strategy and performs multi-step memory reasoning through temporal difference; Constructing a multi-scale query vector for the fused features, the multi-scale query vector includes a spatial scale vector, a temporal scale vector and a channel scale vector, retrieving a memory feature having a maximum cosine similarity with the query vector from a memory module, integrating the fused features with the fused features through an adaptive fusion network, and obtaining a scene adaptive feature through a residual connection; The memory module is optimized based on a weighted combination of memory duration and feature mutual information, including priority updating, memory compression, and dynamic adjustment of memory capacity.

5. The method according to claim 4, wherein Input the memory module into the reinforcement learning framework, which includes an Actor-Critic network structure and a multi-step memory reasoning mechanism. The Actor network generates a memory operation policy based on the current memory content, the fusion feature, and the historical operation sequence. The Critic network uses a dual value evaluation network structure to evaluate the state-action value function. The steps of constructing a reward function based on feature mutual information, distribution distance, and memory timeliness, updating the policy using policy gradients, and performing multi-step memory reasoning through temporal difference include: The Actor network encodes the current memory content through a multi-layer perceptron to obtain a memory representation, performs feature transformation on the fusion feature through a linear transformation to obtain a fusion feature representation, encodes the historical operation sequence through a gated recurrent unit network to obtain a historical representation, concatenates the memory representation, fusion feature representation, and historical representation, and then obtains a memory operation policy distribution through a fully connected layer; Construct a Critic network using a dual value evaluation network, including a feature value network based on feature mutual information and distribution distance, and a memory value network based on storage duration and usage frequency, and obtain the state-action value function by combining the outputs of the feature value network and the memory value network through a learnable balance factor; Construct a comprehensive reward function based on multi-factor coupling, including a mutual information measurement term between features and memories, a distribution distance term between source scene and target scene features, and a memory timeliness measurement term; Set the weight coefficients to be determined through grid search and cross-validation, and dynamically adjust the weights according to the change rates of mutual information, distribution distance, and memory timeliness; Use n-step temporal difference to construct the cumulative reward, and update the Actor network based on the rewards from the current to the next n states and the decay values of the state-action value function; Iteratively update the memory operation policy through the temporal difference error; Introduce a memory tracking mechanism based on importance sampling, calculate the importance weights of each state in the historical state sequence, where the importance weight is the ratio of the current policy probability to the behavior policy probability, and use the cumulative product of the importance weight and the decay factor as the state importance measure to track the memory operation based on the state importance.

6. The method according to claim 1, characterized in that, Input the scene adaptive feature into the multi-task learning network, extract the comprehensive feature through the shared representation layer, output the anomaly prediction result through the task attention layer, and input the anomaly prediction result into the risk propagation neural network. The steps of analyzing the coupling relationship of risk factors and outputting the risk level through the message passing mechanism include: Based on the scene adaptive feature, extract local to global features through the multi-scale convolutional kernels of the shared representation layer of the multi-task learning network, and fuse them through the feature pyramid network to obtain the comprehensive feature; Calculate the attention scores using the multi-head collaborative attention mechanism, integrate the multi-head outputs, and fuse them with the comprehensive feature through the residual gating mechanism to obtain the anomaly prediction result; Input the anomaly prediction result into the risk propagation neural network, construct a risk factor relationship graph, and process the risk factor nodes and the association relationships between factors through the node feature network and the edge feature network respectively to obtain the initial node features and the edge features of the association relationships between risk factors; Adopt a message passing mechanism with gated graph convolution, where the message generation sub-module combines the source node features, target node features, and edge features through a gated unit, and the message aggregation sub-module performs weighted aggregation on the neighborhood message features based on the attention coefficients between nodes; Perform multiple rounds of message passing on the risk factor relationship graph to iteratively update the node features, and add the risk factor node features after a preset number of rounds of message passing to the initial risk factor node features after residual mapping to obtain the risk factor node states; calculate the risk propagation intensity between risk factor nodes based on the feature similarity between risk factor node states and the corresponding edge features, and integrate through hierarchical graph pooling to obtain the risk level.

7. The method according to claim 1, wherein The steps of inputting the risk level into a recurrent neural network, combining historical warning feedback information, and performing feature memory through a gated mechanism and outputting a warning strategy include: Encoding the risk level through a bidirectional long short-term memory network to obtain a fused temporal representation; inputting the historical warning feedback information into a feature transformation network to obtain a feedback feature representation; calculating attention weights at each layer of the time granularity pyramid based on the fused temporal representation and the feedback feature representation, and weighted integration through a soft attention mechanism to obtain multi-granularity memory features; Designing a forgetting gate and an update gate to dynamically update the multi-granularity memory features, where the forgetting gate controls the retention degree of historical information based on the fused temporal representation and the multi-granularity memory features, and the update gate controls the introduction ratio of new knowledge to obtain updated memory features; inputting the fused temporal representation and the updated memory features into a gated recurrent unit for adaptive fusion to obtain a warning state representation; Constructing a multi-objective warning optimization function, which comprehensively considers warning timeliness, warning accuracy, and abnormal behavior recognition rate. Input the warning state representation into a deep Q-learning network, construct a state-action value function based on scene adaptive features and warning response behaviors, where the state of the state-action value function includes scene adaptive features and environmental parameters, and the actions of the state-action value function include warning level adjustment and intervention measure recommendation. Map the state-action value function to obtain a warning strategy, and iteratively optimize the warning strategy through a temporal difference algorithm to obtain an optimal warning strategy.

8. A home abnormal state signal detection system based on multimodal sensing, for implementing the method described in any one of the foregoing claims 1-7, characterized in that, Include: The first unit is used to input the collected human motion data, acoustic features, and environmental parameters into a deep fusion network. The deep fusion network performs dynamic weight allocation on different features through a spatio-temporal attention mechanism, extracts spatio-temporal feature sequences through a multi-scale convolutional neural network, and combines them with the spectral analysis results of acoustic features to form joint probability features; The second unit is used to process the joint probability features through the temporal knowledge graph of the graph attention network to obtain multi-dimensional combined features, input the multi-dimensional combined features into a two-stream autoencoder network, perform feature mapping on the source scene, and use adversarial training to obtain target scene features; Input the source scene features and target scene features into a memory-augmented neural network, and output scene adaptive features in combination with a reinforcement learning method; The third unit is used to input the scene adaptive features into a multi-task learning network, extract comprehensive features through a shared representation layer, output an anomaly prediction result through a task attention layer, input the anomaly prediction result into a risk propagation neural network, analyze the coupling relationship of risk factors through a message passing mechanism and output a risk level; input the risk level into a recurrent neural network, perform feature memory through a gating mechanism in combination with historical warning feedback information and output a warning strategy.

9. An electronic device, characterized in that, It includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images