A smart home instruction understanding method based on fusion of voiceprint and environmental context

Abstract

This invention proposes a smart home command understanding method based on the fusion of voiceprint and environmental context. First, by dynamically constructing a contextual mental graph and using graph neural network inference, it models multi-source heterogeneous information from the home and generates high-level contextual embeddings, overcoming the limitations of single dimensions and improving the accuracy of implicit or fuzzy intent recognition. Second, it introduces an intent-behavior probability potential field and combines it with a generator module based on generative adversarial networks to dynamically model the joint probability distribution of user intent and system behavior. This, combined with a deep reinforcement learning module, enables refined demand prediction and proactive device operation. Finally, through contextual perturbation and causal counterfactual inference modules, it quantifies the causal relationship between key contextual variables and the probability potential field, optimizes the causal edge weights of the graph, avoids erroneous decisions, and enhances the system's generalization ability and decision reliability.

Description

Technical Field

[0001] This invention relates to the field of smart home data processing technology, and in particular to a smart home command understanding method based on the fusion of voiceprint and environmental context. Background Technology

[0002] Smart home technology, especially intelligent control systems centered on voice interaction, has been widely used in modern living environments. Current mainstream smart home systems typically rely on Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) technologies to interpret users' explicit voice commands, thereby controlling corresponding smart devices, such as turning lights on and off, adjusting air conditioning temperature, or playing music. These systems demonstrate high execution efficiency when processing structured and explicit commands.

[0003] As users' demands for intelligent and personalized services continue to rise, the limitations of existing technological solutions are becoming increasingly apparent. A significant shortcoming of current command understanding methods lies in their limited ability to perceive and integrate environmental context. A user's true intent is often deeply embedded in their specific situation, defined by multiple dimensions including time, environmental state, current device state, user's historical behavior, and even the user's physiological or emotional state. Existing technologies typically process voice commands in isolation, and even when combined with some contextual information, they mostly remain at the level of simple rules, lacking mechanisms for deep integration and dynamic reasoning of multi-source heterogeneous information. This leads to difficulties in accurate judgment and response when user commands are ambiguous, omitted, or rely entirely on implicit expressions of the current context.

[0004] Furthermore, existing smart home systems generally rely on statistical correlation analysis of historical data for decision-making logic, rather than a deep understanding of causal relationships. These systems attempt to predict user needs or automate operations by learning the superficial correlations between user behavior and specific scenarios. This correlation-based decision-making model is particularly vulnerable to new and unseen combinations of scenarios and cannot adapt to dynamic changes in user habits. For example, a system may learn the behavioral pattern of "user A turns on the lights at 7 pm," but it cannot understand the underlying causal relationship that "darkness leads to the need for lighting." Therefore, when external conditions change (such as seasonal changes causing earlier darkness), the system cannot make adaptive adjustments, fundamentally limiting its proactive service capabilities and decision-making robustness. Therefore, how to construct a smart home command understanding method that can deeply integrate dynamic scenarios, understand the causal logic behind user intentions, and make proactive and accurate decisions accordingly has become a pressing technical problem in this field. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] Therefore, the first objective of this invention is to propose a smart home instruction understanding method based on the fusion of voiceprint and environmental context.

[0007] The second objective of this invention is to propose a smart home command understanding device based on the fusion of voiceprint and environmental context.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a method for understanding smart home commands based on the fusion of voiceprint and environmental context, comprising: S1. Collect and preprocess multimodal data, including voiceprint data, environmental sensor data, smart device status data, user behavior logs and external information, and generate a synchronous time series data stream; S2, construct a scenario mental graph containing multiple types of nodes and dynamic weighted edges, use a graph neural network to infer scenario embeddings from the scenario mental graph, and dynamically update the edge weights of the scenario mental graph based on the causal counterfactual reasoning results. S3, cross-modal temporal fusion of the synchronous time series data stream and the scenario embedding is performed through a multi-head attention mechanism to generate a fused multimodal scenario representation; S4. Based on the fusion of multimodal scenario representation and causal counterfactual reasoning results, a deep reinforcement learning Actor-Critic architecture is used to perform causal-driven intent inference and intelligent decision-making, generating optimal smart home device operation instructions.

[0009] In one embodiment of the present invention, S2 includes: S21. Construct a multi-type node graph that includes person nodes, environment status nodes, device status nodes, time nodes, external information nodes, and abstract intent nodes. Among them, person nodes represent the identity characteristics of family members, environment status nodes represent environmental parameters such as temperature, humidity, light intensity, and air quality, and device status nodes represent the real-time operating status of smart home devices. S22, the weight update of dynamic weighted edges is achieved through three approaches: initial weight allocation based on the co-occurrence frequency of nodes statistically analyzed in user behavior logs; adjustment of preference relation edge weights based on user feedback signals output by the deep reinforcement learning module; and updating of causal relation edge weights based on the KL divergence quantification results of the causal counterfactual reasoning module.

[0010] In one embodiment of the present invention, S3 includes: S31, the calculation in the multi-head attention mechanism is as follows: , in, , , These represent the query matrix, key matrix, and value matrix, respectively. Indicates a splicing operation; Indicates the number of attention heads; This represents the learnable output projection matrix; Indicates the first The output of each attention head is calculated using the following formula: ; S32, During the cross-modal fusion process, position encoding is applied to all feature sequences respectively. The position encoding adopts a weighted combination of a fixed position vector generated by a sine-cosine function and a learnable position vector.

[0011] In one embodiment of the present invention, S4 includes: S41, in the Actor-Critic architecture, construct a state space containing the current fused multimodal context representation, CCG context embedding, representation of the highest probability intent-behavior pair in IAPPF, and vectors of the system's internal memory states. S42, the reward function design includes three dimensions: positive reward when the user does not manually intervene, negative reward when the user manually corrects, and sparse reward for the matching degree of the key causal chain identified by the system's proactive behavior and the causal counterfactual reasoning module.

[0012] In one embodiment of the present invention, it further includes: S5 verifies the causal chain of the optimal smart home device operation instructions. By querying the causal relationship edge weights in the scenario mental graph, the causal chain with a matching degree higher than the preset threshold with the current scenario embedding is selected as the operation basis. S6 feeds back the verified causal chain to the user terminal in a visual form, and transforms the causal chain into an interpretable decision description through the natural language generation module.

[0013] To achieve the above objectives, a second aspect of the present invention provides a smart home command understanding device based on the fusion of voiceprint and environmental context, comprising: The multimodal data acquisition and preprocessing module is used to collect and preprocess multimodal data, including voiceprint data, environmental sensor data, smart device status data, user behavior logs and external information, and generate a synchronous time series data stream. The context mental graph construction and dynamic update module is used to construct a context mental graph containing multiple types of nodes and dynamically weighted edges, use a graph neural network to infer context embeddings from the context mental graph, and dynamically update the edge weights of the context mental graph based on the causal counterfactual reasoning results. The cross-modal temporal fusion module is used to perform cross-modal temporal fusion of the synchronous time series data stream and the scenario embedding through a multi-head attention mechanism to generate a fused multimodal scenario representation; The causal-driven intent inference and decision-making module is used to perform causal-driven intent inference and intelligent decision-making based on the fusion of multimodal scenario representation and causal counterfactual reasoning results, and to generate optimal smart home device operation instructions using a deep reinforcement learning Actor-Critic architecture.

[0014] The beneficial effects of this invention are as follows: 1. This invention achieves structured and dynamic modeling of multi-source heterogeneous information (people, environment, devices, time, and external information) within a family through the dynamic construction of Contextual Mind Maps (CCG) and inference using Graph Neural Networks (GNN), generating high-level contextual embeddings from this data. This technical feature enables the system to transcend the single-dimensional limitations of traditional sensor data or voice commands, extracting deep contextual information from complex relationships and improving the accuracy of recognizing implicit or ambiguous intentions of users in specific contexts.

[0015] 2. This invention introduces the Intent-Behavior Probabilistic Potential Field (IAPPF) and combines it with a generator module based on the Generative Adversarial Network (GAN) concept. This technique dynamically models the joint probability distribution of user latent intentions and system behaviors in a continuous multidimensional space, enabling the system to understand the ambiguity and multiple possibilities of intentions. The DRL module can explore actions within the high-probability region indicated by the IAPPF, allowing the system to transcend discrete intention classification and achieve more refined predictions of user needs and more proactive, context-adaptive device operation.

[0016] 3. Through scenario perturbation and causal counterfactual reasoning modules, the causal relationship between key scenario variables and the IAPPF is quantified and learned, and the causal edge weights in CCG are dynamically optimized accordingly. This technical feature enables the system to move from the data association level to the causal understanding level, thereby avoiding erroneous decisions based on superficial correlations. Through the simulation and analysis of counterfactual scenarios, when faced with unknown or ambiguous scenarios, the system can make more robust and interpretable decisions based on causal inference, further enhancing the system's generalization ability and decision reliability.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1This is a flowchart of a smart home instruction understanding method based on the fusion of voiceprint and environmental context according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a smart home command understanding method based on the fusion of voiceprint and environmental context according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the data acquisition and preprocessing module according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a contextual mind mapping module according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a multimodal temporal scene representation fusion module according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the intent-behavior probability potential field module according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the causal counterfactual reasoning and CCG optimization module according to an embodiment of the present invention; Figure 8 This is a structural diagram of a smart home command understanding device based on the fusion of voiceprint and environmental context according to an embodiment of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] The following description, with reference to the accompanying drawings, describes a smart home command understanding method and apparatus based on the fusion of voiceprint and environmental context, according to an embodiment of the present invention.

[0022] Figure 1 This is a flowchart of a smart home command understanding method based on the fusion of voiceprint and environmental context according to an embodiment of the present invention, including: S1. Collect and preprocess multimodal data, including voiceprint data, environmental sensor data, smart device status data, user behavior logs and external information, and generate a synchronous time series data stream; S2, construct a scenario mental graph containing multiple types of nodes and dynamic weighted edges, use a graph neural network to infer scenario embeddings from the scenario mental graph, and dynamically update the edge weights of the scenario mental graph based on the causal counterfactual reasoning results. S3, cross-modal temporal fusion of the synchronous time series data stream and the scenario embedding is performed through a multi-head attention mechanism to generate a fused multimodal scenario representation; S4. Based on the fusion of multimodal scenario representation and causal counterfactual reasoning results, a deep reinforcement learning Actor-Critic architecture is used to perform causal-driven intent inference and intelligent decision-making, generating optimal smart home device operation instructions.

[0023] This invention presents a smart home command understanding method based on the fusion of voiceprint and environmental context, which possesses three major technical advantages and can significantly improve the performance of home smart systems. First, it breaks through the limitations of a single dimension by modeling multi-source heterogeneous information through contextual mental graphs to extract deep context, greatly improving the recognition accuracy of implicit and fuzzy user intentions. Second, it abandons discrete classification and relies on intention-behavior probability potential fields and deep reinforcement learning to achieve refined demand prediction and proactive device operation adaptation. Third, it upgrades to the causal understanding level, using causal counterfactual reasoning to avoid misjudgments of associations, enhancing the system's robustness, interpretability, and generalization ability in unknown scenarios.

[0024] The following describes in detail an embodiment of the present invention, based on the fusion of voiceprint and environmental context, for understanding smart home commands according to the accompanying drawings.

[0025] This invention provides a method for understanding smart home commands based on the fusion of voiceprint and environmental context, including the following steps: Figures 2 to 7 As shown: S10. Data Acquisition and Preprocessing. This step is performed by the data acquisition and preprocessing module. This module acquires multimodal data through multiple sensors and interfaces, including voiceprint data, environmental sensor data, smart device status data, user behavior logs, and external information. The module cleans, normalizes, and synchronizes the acquired data to generate a time-series data stream in a unified format, and outputs it to the contextual mind mapping module and the multimodal time-series contextual representation fusion module.

[0026] S20. Dynamic Construction of Contextual Mental Graph and Generation of Context Embedsions. This step is performed by the Contextual Mental Graph module. This module receives preprocessed multimodal data provided by the data acquisition and preprocessing module, and uses it to construct and dynamically update a Contextual Mental Graph (CCG) containing multiple types of nodes and dynamically weighted edges. Subsequently, this module uses a Graph Neural Network (GNN) to infer the CCG and generate context embeddings for the graph at the current time step. In the specific implementation, the hierarchical update calculation of the GNN can follow the following formula:

[0027] in, Indicates the first Feature representation matrix of layer nodes; Indicates the first Feature representation matrix of layer nodes; Represents a non-linear activation function; This indicates that the adjacency matrix with added self-loops has been added, i.e. ,in This is the original adjacency matrix. It is the identity matrix; express A degree matrix, where the diagonal elements ; Indicates the first A learnable weight matrix for each layer. The contextual mind mapping module embeds the generated context into the multimodal temporal contextual representation fusion module.

[0028] S30. Multimodal Temporal Scene Representation Fusion. This step is performed by the multimodal temporal scene representation fusion module 3. This module receives preprocessed multimodal data from the data acquisition and preprocessing module and scene embeddings from the scene mind map module. Internally, this module employs a Transformer encoder based on a multi-head attention mechanism to perform deep temporal fusion on multi-source inputs, including encoded voiceprint features, environmental / device / behavioral feature sequences, and scene embedding sequences. In the specific implementation, the multi-head attention calculation follows the formula:

[0029] in, , , These represent the query matrix, key matrix, and value matrix, respectively. Indicates a splicing operation; Indicates the number of attention heads; This represents the learnable output projection matrix; Indicates the first The output of each attention head is calculated using the following formula:

[0030] in , , , for the first The learnable projection matrix corresponding to each size. The key dimension. This module ultimately generates a fused multimodal scenario representation and outputs it to the intention-behavior probability potential field module and the scenario perturbation and causal counterfactual reasoning module, respectively.

[0031] S40. Dynamic Modulation of the Intent-Behavior Probability Potential Field. This step is performed by the Intent-Behavior Probability Potential Field module. This module receives the fused multimodal scenario representation from the multimodal temporal scenario representation fusion module and constructs and dynamically modulates the Intent-Behavior Probability Potential Field (IAPPF) based on it. The IAPPF is a dynamic, multidimensional probability distribution used to model the joint probability density of the user's potential intent and system behavior in the current scenario. This module outputs the high-probability region information of intent-behavior indicated by the IAPPF to the Intent Inference and Intelligent Decision-Making module.

[0032] S50, Causal Counterfactual Reasoning and CCG Optimization. This step is performed by the Context Perturbation and Causal Counterfactual Reasoning module. This module receives fused multimodal context representations from the modal temporal context representation fusion module and interacts with the Contextual Mental Graph module. Through context perturbation and causal counterfactual reasoning, this module identifies and quantifies the impact of key contextual variables on IAPPF. This module outputs the quantified causal analysis results to the Contextual Mental Graph module to update the edge weights of causal relationships in the CCG; and to the Intent Inference and Intelligent Decision-Making module to provide causal guidance.

[0033] S60, Deep Reinforcement Learning Decision Making. This step is performed by the Intent Inference and Intelligent Decision Making module. This module receives IAPPF high-probability region information from the Intent-Behavior Probability Potential Field module and causal analysis results from the Context Perturbation and Causal Counterfactual Reasoning module. Based on the Actor-Critic architecture of Deep Reinforcement Learning (DRL), this module performs intent inference and intelligent decision making to select the optimal smart home device operation.

[0034] S70, Command Conversion and Device Execution. This step is performed by the behavior execution module. This module receives the optimal smart home device operation from the intent inference and intelligent decision-making module, converts it into control commands that the target smart home device can recognize, and sends them to the target device through the communication interface to complete the execution.

[0035] In step S10, the data acquisition and preprocessing module performs the acquisition and preprocessing of multi-modal odal data, such as... Figure 3As shown. In a specific embodiment, voiceprint data is collected through a microphone array deployed indoors; environmental sensor data is collected by temperature and humidity sensors, light intensity sensors, and PM2.5 sensor devices; smart device status data is directly obtained through communication interfaces with various smart home devices (such as air conditioners, lights, and curtains); user behavior logs record all historical interactions between the user and the system; external information is obtained from third-party services (such as weather forecasts and calendar services) through network interfaces.

[0036] During the preprocessing of the collected data, missing or outlier data points are first interpolated or removed. Then, numerical data of different dimensions, such as temperature and light intensity, are scaled to a uniform range using min-max normalization. Finally, all data streams are marked with precise timestamps and aligned to fixed time intervals. The time windows are divided to form synchronized time-series data for subsequent modules to process.

[0037] In step S20, the Contextual Mind Mapping module performs the construction, reasoning, and dynamic updating of the Contextual Mind Map (CCG), such as... Figure 4 As shown, a multi-type node graph is constructed, comprising person nodes, environment state nodes, device state nodes, time nodes, external information nodes, and abstract intent nodes. Person nodes represent family member identity characteristics, environment state nodes represent environmental parameters such as temperature, humidity, light intensity, and air quality, and device state nodes represent the real-time operating status of smart home devices. Dynamic weighted edge updates are achieved through three approaches: initial weight allocation based on node co-occurrence frequency statistics from user behavior logs; adjustment of preference relation edge weights based on user feedback signals output by the deep reinforcement learning module; and updating of causal relation edge weights based on the KL divergence quantification results from the causal counterfactual reasoning module.

[0038] In a specific embodiment, CCG nodes are divided into: person nodes representing family members. ; Environment state nodes representing the environmental state Device status nodes represent the operating status of the equipment. ; Time nodes representing time attributes External information nodes representing external events ; and abstract intent nodes representing higher-level user intents. Weighted edges between nodes represent specific relationships between them, such as causal, preference, or co-occurrence relationships.

[0039] This module employs a Graph Neural Network (GNN) for inference on the CCG. In a specific implementation, the GNN model is a Graph Convolutional Network (GCN). The GCN updates the feature representation of each node by aggregating neighbor node information layer by layer. Its hierarchical update calculation follows the formula:

[0040] in, It is the first Feature representation matrix of all nodes in the layer; It is the first The feature representation matrix of all nodes in the layer, and This is the initial feature matrix of the node; It is a non-linear activation function, such as the ReLU function; It is a graph adjacency matrix with added self-loops, i.e. ,in This is the original adjacency matrix. It is the identity matrix; yes A degree matrix, where the diagonal elements ; It is GCN's first Layer-learnable weight matrices. After multi-layer GCN processing, each node obtains a high-dimensional vector representation that integrates graph structure and neighbor information, i.e., context embedding. .

[0041] CCG's dynamic weighted edge weights Learning and adjustment are achieved through three approaches: 1) Based on user behavior logs, through statistical nodes and nodes 1) Update the edge weights of the CCG based on the co-occurrence frequency within the time window; 2) Adjust the edge weights of the CCG based on the user feedback (reward or punishment signal) obtained from the decision output by the intent inference and intelligent decision-making module; 3) Update the edge weights of the corresponding causal relationship in the CCG directly based on the quantitative causal effect output by the scenario perturbation and causal counterfactual reasoning module.

[0042] In step S30, the multimodal temporal scene representation fusion module performs the fusion of multimodal features, such as... Figure 5 As shown, during the cross-modal fusion process, positional encoding is applied to all feature sequences. The positional encoding adopts a weighted combination of a fixed position vector generated by a sine-cosine function and a learnable position vector.

[0043] In a specific embodiment, the input voiceprint data is first encoded using a pre-trained self-supervised speech model (e.g., wav2vec2.0) to extract the user identity embedding that represents the user's identity. Emotional acoustic feature embeddings that represent emotional information in speech (such as speech rate, pitch, and energy) Environmental, equipment, and behavioral data are then constructed into time-series feature sequences.

[0044] This module employs a multi-layer Transformer encoder structure to process all input feature sequences. , Environmental / equipment / behavioral characteristic sequences and The Transformer encoder fuses sequences. It captures the complex dependencies within a sequence and between different modal sequences through its core multi-head self-attention mechanism. Its multi-head attention calculation follows this formula:

[0045] in, Here, , , These represent the query, key, and value matrices, respectively, which are obtained from the embeddings of the input sequences through linear transformations. , , and It is a learnable projection matrix; It is the dimension of the key; This refers to the number of attention heads. The encoder also includes positional encoding to provide positional information for elements in the sequence. The module ultimately outputs a high-dimensional fused multimodal scene representation. .

[0046] In step S40, the intent-behavior probability potential field module performs the construction and modulation of the intent-behavior probability potential field (IAPPF), such as... Figure 6 As shown, the Actor-Critic architecture constructs a state space containing vectors representing the current fused multimodal context, CCG context embedding, the highest probability intent-behavior pair representation in IAPPF, and the system's internal memory states. The reward function design includes three dimensions: positive reward when the user does not manually intervene, negative reward when the user manually corrects, and sparse reward based on the matching degree of the key causal chain identified by the system's proactive behavior and the causal counterfactual reasoning module.

[0047] In a specific embodiment, IAPPF is defined as the user's latent intent vector in the current context. With the system's potential behavior vector joint probability density function This module contains a modulation network. It receives fused multimodal scenario representations As input, the output parameters define the IAPPF probability distribution (e.g., a Gaussian mixture model) at the current time step. (Mean, covariance, and weight of the quantities).

[0048] To generate intent-behavior pairs from IAPPF, this module also includes a generator module based on the concept of Generative Adversarial Networks (GANs). Generator Receive random noise and current situation representation As input, synthesized intent-behavior pairs are generated. Discriminator This is used to determine whether the input intent-behavior pair comes from real historical data or is generated by the generator. Synthesis. The generator is optimized through adversarial training. This enables it to generate samples that conform to the probability distribution indicated by the current IAPPF.

[0049] In step S50, the scenario perturbation and counterfactual reasoning module performs quantification of causal effects, such as... Figure 7 As shown, the optimal smart home device operation commands are verified through causal chain validation. By querying the causal relationship edge weights in the scenario mental graph, causal chains with a matching degree higher than a preset threshold with the current scenario embedding are selected as the basis for operation.

[0050] In a specific embodiment, the module first identifies the fused multimodal scene representation of the final output by analyzing the attention weights of the Transformer encoder in the multimodal temporal scene representation fusion module. The key contextual variables that contribute the most (e.g., cough features in voiceprints or PM2.5 values ​​in environmental data).

[0051] Subsequently, the key situational variables identified were... This module applies intervention operations, for example, to The value is fixed as a counterfactual value. And generate counterfactual scenario representations. By comparing the IAPPF distribution in real-world scenarios. IAPPF distribution under counterfactual scenarios Quantification variables The causal effect. In a specific implementation, this quantization is achieved by calculating the KL divergence between two probability distributions. To complete.

[0052] In step S60, the intent inference and intelligent decision-making module performs deep reinforcement learning decision-making. In a specific embodiment, this module adopts an Actor-Critic architecture. Its state space... Defined as including the current fused multimodal scenario representation CCG Context Embedding The highest probability in IAPPF generated by the intention-behavior probability potential field module The representation of each intention-behavior pair, and the internal memory state of the system. The vector. Its action space. Includes all executable discrete or continuous device operations, such as adjusting light brightness. Set the air conditioner temperature .

[0053] Actor Network Receive current state The output is the probability distribution in the action space, and its parameters are... Update by maximizing expected returns. Critic Network Receive current state and the action chosen by the Actor Predict the expected cumulative return that can be obtained after performing this action, its parameters Updates are performed by minimizing the temporal difference error. Reward function. The design includes: providing positive rewards when the user's behavior (such as no manual intervention) indicates acceptance of the system's proactive operation; providing negative rewards when the user manually corrects the system's operation; and providing additional sparse rewards when the system's proactive behavior matches the contextual perturbation and the key causal chain identified by the causal counterfactual reasoning module and is not corrected by the user.

[0054] In step S70, the behavior execution module performs instruction conversion and device execution. The verified causal chain is fed back to the user terminal in a visual form, and the natural language generation module transforms the causal chain into an interpretable decision description.

[0055] This module receives abstract actions representing the optimal operation (e.g., from the intent inference and intelligent decision-making module). The module then converts these commands into standardized control instructions that conform to the communication protocols of specific smart home devices (such as Zigbee, Wi-Fi, or Matter). The module then sends these instructions to the target device via the corresponding communication interface to execute the physical operation.

[0056] To more comprehensively define the scope of protection of this invention, it should be noted that those skilled in the art will understand that, without departing from the core idea of ​​this invention, there are various alternative implementations of some or all of the technical solutions described in this invention. Although these alternatives differ in specific implementation details, device selection, or algorithm paths, their ultimate purpose is the same as the technical problem to be solved and the beneficial effects achieved by this invention.

[0057] I. Alternatives to "Contextual Mind Mapping (CCG)": This invention employs dynamic graph structures (CCG) and graph neural networks (GNN) to represent and infer contextual information. Any technique capable of structurally representing multi-source heterogeneous information and performing reasoning can be considered an equivalent alternative to this approach.

[0058] An ontology-based knowledge base approach can forgo a flexible graph structure and instead construct a formalized ontology-based knowledge base for the smart home domain (e.g., using languages ​​like OWL or RDF). In this approach, entities (users, devices, locations), attributes, and the logical relationships between them (e.g., isA, hasProperty, locatedIn) are rigorously defined. Instead of a GNN, a Reasoning Engine will be used, which infers the current situation and generates situational representations through symbolic logical reasoning. This approach offers stronger logical interpretability but relatively weaker adaptive learning capabilities.

[0059] Implicit context representation schemes based on tensor / matrix factorization: Instead of constructing an explicit graph structure, all context variables (user, time, device, behavior, etc.) are treated as different dimensions, forming a high-dimensional context tensor. This sparse interaction tensor is then decomposed into multiple low-dimensional latent vectors using tensor factorization techniques (such as Tucker decomposition and PARAFAC) or matrix factorization. The combination of these latent vectors constitutes an implicit representation of the current context (i.e., context embedding), thus replacing the step of generating context embeddings using GNNs.

[0060] A semantic association scheme based on a vector database can embed all collected discrete contextual information (such as "User A", "living room", "lights on", "8 PM") into high-dimensional vectors using a pre-trained model. These vectors are then stored in a vector database. The current contextual representation can be generated by querying a series of vectors related to the current moment and aggregating (e.g., weighted averaging) these vectors. This approach implicitly replaces the edge relationships in CCG with semantic proximity between vectors.

[0061] II. Alternative solutions for the "multimodal temporal context representation fusion module": This invention employs a Transformer-based fusion architecture. Any model that can effectively fuse heterogeneous, time-series data streams and capture their complex dependencies can be considered an equivalent alternative to this solution.

[0062] A fusion scheme based on recurrent neural networks (RNNs) can be implemented using parallel long short-term memory networks (LSTMs) or gated recurrent units (GRUs) to process different data modalities (voiceprint feature sequences, environmental feature sequences, etc.). Then, the hidden states of each RNN at each time step are concatenated and fed into another top-level RNN (or an RNN with an attention mechanism) for final fusion, generating a fused contextual representation.

[0063] A fusion scheme based on cross-modal attention mechanisms can be adopted instead of using the complete Transformer self-attention structure. Instead, specific cross-modal attention modules can be designed. For example, the processed voiceprint feature sequence can be used as the "query," and the environmental / device state sequence can be used as the "key" and "value," thereby calculating the association weight between acoustic events and environmental states and achieving targeted information fusion.

[0064] LateFusion approach: Each single-modal data stream can be independently processed by its own network (such as CNN or RNN) to extract high-level features. For example, preliminary "intent classification" can be extracted from speech, and "scene classification" can be extracted from the environment. Finally, before the decision layer, these high-level feature vectors from different modalities are simply concatenated as the final fused representation.

[0065] III. Alternatives to the Intention-Behavior Probability Potential Field (IAPPF): This invention employs a dynamically modulated probability potential field to model the joint distribution of intent and behavior. Any technique capable of outputting a probability distribution or candidate set of future possibilities (intent or behavior) based on the current situation can be considered an equivalent alternative to this solution.

[0066] A classification / regression scheme based on multi-task learning can model the problem as a multi-task learning model. The model's input is a fused context representation, and its output consists of multiple parallel prediction heads: one head classifies the user's core intent (N-classification problem), another predicts the target device of the operation (M-classification problem), and a third regresses specific operational parameters (such as volume, light intensity, etc.). The combination of all these prediction results replaces the sampling process from IAPPF.

[0067] A generative scheme based on variational autoencoders (VAEs) can be used: a conditional variational autoencoder (CVAE) can be employed. The encoder maps the fused contextual representation to a probability distribution (e.g., a Gaussian distribution) in a latent space, which represents the user's latent intent. The decoder then samples from this latent space to generate specific, probable sequences of user behavior.

[0068] IV. Alternatives to Deep Reinforcement Learning (DRL) Decision Making: This invention employs an Actor-Critic framework for decision-making. Any optimization algorithm capable of making sequential decisions using contextual information and feedback signals can be considered an equivalent replacement for this approach.

[0069] Planning-based decision-making: The smart home environment can be modeled as a partially observable Markov decision process (POMDP). The system's state is defined by a scenario representation and a probability distribution (an alternative to IAPPF). Then, an online planning algorithm (such as MCTS, Monte Carlo Tree Search) is used to search forward at each step of the decision, evaluating the long-term value of different action sequences and selecting the optimal initial action.

[0070] Imitation Learning-based decision-making schemes can begin by collecting a large amount of "expert data," which consists of records of ideal human actions in various scenarios. Then, a policy network is directly trained using supervised learning methods such as behavioral cloning, enabling it to mimic the decision-making behavior of experts. This method can serve as a pre-training stage for DRL (Device Decision Learning) or as a standalone, simpler decision-making scheme.

[0071] In summary, regardless of whether symbolic logic reasoning, other types of neural network architectures, or different probabilistic modeling and decision optimization algorithms are used, as long as the technical solution contains the core innovative concept of "dynamically collecting and structurally representing multimodal scenario information, inferring the probability distribution of the user's potential intentions based on the scenario information, and using the distribution and environmental feedback to make closed-loop optimization decisions," it should be considered to fall within the protection scope of this invention.

[0072] To implement the above embodiments, such as Figure 8 As shown, this embodiment also provides a smart home instruction understanding device 10 based on the fusion of voiceprint and environmental context. The device 10 includes a multimodal data acquisition and preprocessing module 100, a contextual mental map construction and dynamic update module 200, a cross-modal temporal fusion module 300, and a causal-driven intent inference and decision-making module 400.

[0073] The multimodal data acquisition and preprocessing module 100 is used to collect and preprocess multimodal data, including voiceprint data, environmental sensor data, smart device status data, user behavior logs and external information, and generate a synchronous time series data stream. The context mental graph construction and dynamic update module 200 is used to construct a context mental graph containing multiple types of nodes and dynamic weighted edges, use a graph neural network to infer context embeddings from the context mental graph, and dynamically update the edge weights of the context mental graph based on the causal counterfactual reasoning results. The cross-modal temporal fusion module 300 is used to perform cross-modal temporal fusion of the synchronous time series data stream and the scenario embedding through a multi-head attention mechanism to generate a fused multimodal scenario representation; The causal-driven intent inference and decision-making module 400 is used to perform causal-driven intent inference and intelligent decision-making based on the fusion of multimodal scenario representation and causal counterfactual reasoning results, and to generate optimal smart home device operation instructions using a deep reinforcement learning Actor-Critic architecture.

[0074] Furthermore, the aforementioned scenario-based mental mapping construction and dynamic update module 200 is also used for: Construct a multi-type node graph that includes person nodes, environment status nodes, device status nodes, time nodes, external information nodes, and abstract intent nodes. Among them, person nodes represent the identity characteristics of family members, environment status nodes represent environmental parameters such as temperature, humidity, light intensity, and air quality, and device status nodes represent the real-time operating status of smart home devices. The weights of dynamically weighted edges are updated through three approaches: initial weight allocation is performed based on the co-occurrence frequency of nodes in user behavior log statistics; the weights of preference relationship edges are adjusted based on user feedback signals output by the deep reinforcement learning module; and the weights of causal relationship edges are updated based on the KL divergence quantification results of the causal counterfactual reasoning module.

[0075] Furthermore, the aforementioned cross-modal timing fusion module 300 is also used for: Configure the calculation in the multi-head attention mechanism , in, , , These represent the query matrix, key matrix, and value matrix, respectively. Indicates a splicing operation; Indicates the number of attention heads; This represents the learnable output projection matrix; Indicates the first The output of each attention head is calculated using the following formula: ; In the cross-modal fusion process, positional encoding is applied to all feature sequences. The positional encoding adopts a weighted combination of fixed position vectors generated by sine-cosine functions and learnable position vectors.

[0076] Furthermore, the aforementioned causal-driven intention inference and decision-making module 400 is also used for: In the Actor-Critic architecture, a state space is constructed that includes the current fused multimodal context representation, CCG context embedding, representation of the highest probability intent-behavior pair in IAPPF, and vectors of the system's internal memory states. The reward function is designed with three dimensions: positive reward when the user does not manually intervene, negative reward when the user manually corrects the behavior, and sparse reward based on the matching degree between the system's proactive behavior and the key causal chain identified by the counterfactual reasoning module.

[0077] Furthermore, the device 10 also includes: The causal chain verification module is used to verify the causal chain of the operation instructions of the optimal smart home device. By querying the causal relationship edge weights in the scenario mental graph, it filters out causal chains with a matching degree higher than a preset threshold with the current scenario embedding as the basis for operation. The visualization feedback module is used to provide the verified causal chain to the user terminal in a visual form, and the natural language generation module transforms the causal chain into an interpretable decision description.

[0078] This invention discloses a smart home command understanding device based on the fusion of voiceprint and environmental context, which possesses three major technical advantages that significantly improve the performance of home smart systems. First, it breaks through the limitations of a single dimension by modeling multi-source heterogeneous information through contextual mental graphs, extracting deep context, and greatly improving the accuracy of recognizing implicit and fuzzy user intentions. Second, it abandons discrete classification and relies on intention-behavior probability potential fields and deep reinforcement learning to achieve refined demand prediction and proactive device operation adaptation. Third, it upgrades to the causal understanding level, using causal counterfactual reasoning to avoid misjudgments of associations, enhancing the system's robustness, interpretability, and generalization ability in unknown scenarios.

[0079] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0080] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for smart home instruction understanding based on fusion of voiceprint and environmental context, characterized in that, include: S1. Collect and preprocess multimodal data, including voiceprint data, environmental sensor data, smart device status data, user behavior logs and external information, and generate a synchronous time series data stream; S2, construct a context mental graph (CCG) containing multiple types of nodes and dynamically weighted edges, use a graph neural network to infer context embeddings from the context mental graph, and dynamically update the edge weights of the context mental graph based on the causal counterfactual reasoning results. S3, cross-modal temporal fusion of the synchronous time series data stream and the scenario embedding is performed through a multi-head attention mechanism to generate a fused multimodal scenario representation; S4. Based on the fusion of multimodal scenario representation and causal counterfactual reasoning results, a deep reinforcement learning Actor-Critic architecture is used to perform causal-driven intent inference and intelligent decision-making to generate optimal smart home device operation instructions. The S2 further includes: S21. Construct a multi-type node graph that includes person nodes, environment status nodes, device status nodes, time nodes, external information nodes, and abstract intent nodes. Among them, person nodes represent the identity characteristics of family members, environment status nodes represent environmental parameters such as temperature, humidity, light intensity, and air quality, and device status nodes represent the real-time operating status of smart home devices. S22, the weight update of dynamic weighted edges is achieved through three approaches: initial weight allocation based on the co-occurrence frequency of nodes statistically analyzed from user behavior logs; adjustment of preference relation edge weights based on user feedback signals output by the deep reinforcement learning module; and updating of causal relation edge weights based on the KL divergence quantification results from the causal counterfactual reasoning module. The S4 further includes: S41, In the Actor-Critic architecture, construct a state space containing the current fused multimodal context representation, CCG context embedding, representation of the highest probability intention-behavior pair in the intention-behavior probabilistic potential field IAPPF, and vectors of the system's internal memory states. S42, the reward function design includes three dimensions: positive reward when the user does not manually intervene, negative reward when the user manually corrects, and sparse reward for the matching degree of the key causal chain identified by the system's proactive behavior and the causal counterfactual reasoning module.

2. The method of claim 1, wherein, The S3 further includes: S31, the calculation in the multi-head attention mechanism is as follows: ， wherein, , , denote the query matrix, the key matrix and the value matrix, respectively; denotes the concatenation operation; denotes the number of attention heads; denotes the learnable output projection matrix; denotes the output of the th attention head, and its calculation formula is ； S32, During the cross-modal fusion process, position encoding is applied to all feature sequences respectively. The position encoding adopts a weighted combination of a fixed position vector generated by a sine-cosine function and a learnable position vector.

3. The method of claim 1, wherein, The method further includes: S5 verifies the causal chain of the optimal smart home device operation instructions. By querying the causal relationship edge weights in the scenario mental graph, the causal chain with a matching degree higher than the preset threshold with the current scenario embedding is selected as the operation basis. S6 feeds back the verified causal chain to the user terminal in a visual form, and transforms the causal chain into an interpretable decision description through the natural language generation module.

4. A smart home instruction understanding device based on fusion of voiceprint and environmental context, characterized in that, include: The multimodal data acquisition and preprocessing module is used to collect and preprocess multimodal data, including voiceprint data, environmental sensor data, smart device status data, user behavior logs and external information, and generate a synchronous time series data stream. The Contextual Mental Graph (CCG) construction and dynamic update module is used to construct a CCG containing multiple types of nodes and dynamically weighted edges. It uses a graph neural network to infer contextual mental graphs to generate contextual embeddings and dynamically updates the edge weights of the CCG based on causal counterfactual reasoning results. The cross-modal temporal fusion module is used to perform cross-modal temporal fusion of the synchronous time series data stream and the scenario embedding through a multi-head attention mechanism to generate a fused multimodal scenario representation; The causal-driven intent inference and decision-making module is used to perform causal-driven intent inference and intelligent decision-making based on the fusion of multimodal scenario representation and causal counterfactual reasoning results, and to generate optimal smart home device operation instructions using a deep reinforcement learning Actor-Critic architecture. The scenario mental map construction and dynamic update module is also used for: Construct a multi-type node graph that includes person nodes, environment status nodes, device status nodes, time nodes, external information nodes, and abstract intent nodes. Among them, person nodes represent the identity characteristics of family members, environment status nodes represent environmental parameters such as temperature, humidity, light intensity, and air quality, and device status nodes represent the real-time operating status of smart home devices. The weight update of dynamic weighted edges is achieved through three methods: initial weight allocation is based on the co-occurrence frequency of nodes according to user behavior log statistics; Adjusting the weights of preference relationship edges based on user feedback signals output by the deep reinforcement learning module; Update the weights of causal relationship edges based on the KL divergence quantization results of the causal counterfactual reasoning module. The causal-driven intent inference and decision-making module is also used for: In the Actor-Critic architecture, a state space is constructed that includes the current fused multimodal context representation, CCG context embedding, representation of the highest probability intent-behavior pair in the intent-behavior probabilistic potential field IAPPF, and vectors of the system's internal memory states. The reward function is designed with three dimensions: positive reward when the user does not manually intervene, negative reward when the user manually corrects the behavior, and sparse reward based on the matching degree between the system's proactive behavior and the key causal chain identified by the counterfactual reasoning module.

5. The apparatus of claim 4, wherein, The cross-modal timing fusion module is also used for: Configure the calculation in the multi-head attention mechanism ， wherein, , , denote the query matrix, the key matrix and the value matrix, respectively; denotes the concatenation operation; denotes the number of attention heads; denotes the learnable output projection matrix; denotes the output of the th attention head, and its calculation formula is ； In the cross-modal fusion process, positional encoding is applied to all feature sequences. The positional encoding adopts a weighted combination of fixed position vectors generated by sine-cosine functions and learnable position vectors.

6. The apparatus of claim 4, wherein, Also includes: The causal chain verification module is used to verify the causal chain of the operation instructions of the optimal smart home device. By querying the causal relationship edge weights in the scenario mental graph, it filters out causal chains with a matching degree higher than a preset threshold with the current scenario embedding as the basis for operation. The visualization feedback module is used to provide the verified causal chain to the user terminal in a visual form, and the natural language generation module transforms the causal chain into an interpretable decision description.

Images