Human-Computer Interaction-Based Smart Home Control Methods and Systems

Abstract

This invention proposes a smart home control method and system based on human-computer interaction. The method includes: collecting user voice signals, gesture images, head posture information, and environmental state information to construct a multimodal interaction sequence; inputting the multimodal interaction sequence into an improved temporal fusion model to identify the user's control intention, outputting a final fusion representation vector, and generating control intention triples through three sets of parallel discrimination modules; evaluating the logical consistency and expression stability of the control intention triples, calculating a confidence score, and combining it with the control intention triples to output an intention quadruple; based on the intention quadruple, when the corresponding confidence score exceeds a preset threshold, converting the corresponding control intention triple into a control command for the corresponding target device and executing it. This invention significantly improves the fuzzy expression parsing capability, the accuracy of target device determination, and the stability and robustness of the actual execution process.

Description

Technical Field

[0001] This invention belongs to the field of smart home control, and particularly relates to a smart home control method and system based on human-computer interaction. Background Technology

[0002] With the increasing prevalence of smart home systems in homes, users' control needs for various home appliances such as lighting, air conditioning, and curtains are becoming more diversified, immediate, and natural. Traditional switch-type and touch-based control methods are no longer sufficient to meet this highly flexible interaction mode. In recent years, multimodal human-computer interaction has gradually become the mainstream direction, and natural expression methods combining voice and gestures have been widely applied in home scenarios to improve user experience and operational efficiency. However, existing technologies still have significant shortcomings in real-world home scenarios. In noisy environments, voice recognition often fails due to background noise, echoes, or multiple people speaking simultaneously, causing the system to be unable to correctly interpret commands. While gesture recognition can serve as a supplement, most methods can only recognize static gestures or a limited number of movements, failing to understand the semantic meaning behind the directionality and amplitude of gestures. Furthermore, users often do not explicitly point to the target device in natural interactions; for example, when facing a light, they may simply say "too bright" accompanied by a downward pressure motion. Existing systems struggle to identify the target direction and cannot infer the desired adjustment range, resulting in unstable control results or false triggering. Meanwhile, user expressions are often vague, non-standard, and incomplete, such as "smaller" or "more like this." These vague instructions cannot be clearly quantified in traditional voice or gesture monomodal recognition, lacking a clear control parameter mapping mechanism. Furthermore, the coexistence of multiple users, fixed device distribution, and frequent operation rhythms in home scenarios further exacerbate the uncertainty of intent recognition and the risk of control conflicts. Existing technologies typically lack an effective mechanism to determine whether the currently recognized interaction intent is worth executing; that is, they lack a reliability filter for the stability and structural consistency of multimodal expressions. This leads to false activations when boundary expressions or signals are incomplete, causing security and user experience issues.

[0003] Therefore, a technical solution that can simultaneously process voice, gestures, gaze direction, and scene context, obtain the user's true intent through temporal modeling and structured semantic inference, and perform reliability judgment before execution to avoid false triggering has become a key problem that smart home control systems urgently need to solve at this stage. Summary of the Invention

[0004] The purpose of this invention is to propose a smart home control method and system based on human-computer interaction to solve the above-mentioned problems.

[0005] To achieve the above objectives, a smart home control method based on human-computer interaction is provided in a first aspect of the present invention, the method comprising the following steps: S1. Collect the user's voice signal, gesture motion image, head posture information and environmental state information to construct a multimodal interaction sequence; wherein, the multimodal interaction sequence contains several modality fusion vectors at a single moment, and the modality fusion vectors include voice features, gesture features, gaze direction features and environmental state features corresponding to the voice signal, gesture motion image, head posture information and environmental state information; S2. Input the multimodal interaction sequence into the improved temporal fusion model to identify the user's control intention and output the final fusion representation vector; based on the final fusion representation vector, generate a control intention triplet containing the target device, operation type and control parameters through three sets of parallel discrimination modules; S3. Evaluate the logical consistency and expression stability of the control intention triplet, calculate the confidence score, and combine it with the control intention triplet to output the intention quadruple. S4. Based on the intent quadruple, when the corresponding confidence score exceeds a preset threshold, the corresponding control intent triple is converted into a control command for the corresponding target device and executed. The three sets of parallel discrimination modules also include a spatial location guidance term and a context adversarial regularization term; the spatial location guidance term is used to guide the direction of the final fused representation vector to align with the user's current gaze device; the context adversarial regularization term is used to reduce target selection interference caused by abnormal environmental offsets.

[0006] Furthermore, the voice signal is acquired through a microphone array; the gesture image is acquired by an RGB-D camera; the head pose information is extracted from facial key points in the same camera image, and a MobileNet-based facial key point estimation algorithm is used to output a three-axis direction cosine code; the environmental state information is provided by the system's internal time module and light sensor, the time period is identified by the system clock, and the light value is the intensity value of the sensor voltage after analog-to-digital conversion and normalization.

[0007] Furthermore, the multimodal interaction sequence is generated as follows: The voice signal, gesture image, head posture information, and environmental state information are uniformly segmented according to the sampling time window; each segment corresponds to an interaction moment, and each modal data is mapped to the range of that moment to form a modality-aligned segment; After alignment is completed, feature extraction and encoding operations are performed on the modal alignment fragments of each modality to obtain the corresponding feature vectors; The corresponding feature vectors are concatenated according to the feature dimension to form a modality fusion vector at a single moment; The modal fusion vectors from consecutive time periods are combined to form a multimodal interaction sequence.

[0008] Furthermore, after alignment is completed, feature extraction and encoding operations are performed on the modal alignment segments of each modality to obtain the corresponding feature vectors, specifically as follows: Mel-spectral signatures were extracted from the speech signal at each time window and processed using a four-layer convolutional network configured with 64 / 128 / 128 / 128 channels and a kernel size of [missing value]. Output a feature vector with a dimension of 128, which is the speech feature; The gesture motion image consists of a coordinate sequence of 21 two-dimensional key points. The curvature change of the key point trajectory is calculated to form a dynamic description of the gesture. After dimensionality reduction, it becomes a 42-dimensional feature vector, which is the gesture feature. The head pose information is used to construct a pose estimation model using 68 facial key points to output the direction angle, and then a three-dimensional direction vector is formed through cosine mapping, which is the gaze direction feature; In the environmental state information, the time period is encoded using three-dimensional one-hot encoding, the illumination value is normalized to a one-dimensional floating-point value, and the two-dimensional environmental features are spliced ​​together to form the environmental state features.

[0009] Furthermore, the improved temporal fusion model consists of a three-layer gated loop structure and a set of modal attention weight matrices, and its output is the final fusion representation vector; wherein, the modal attention weight matrix is ​​used to dynamically emphasize the contribution of different time points and different modalities to the final control intent.

[0010] Furthermore, the spatial location guidance term is obtained by performing an inner product operation on the final fused representation vector and the spatial gaze reference vector corresponding to each target device; the context adversarial regularization term is calculated by combining the final fused representation vector and the current environment state projection vector.

[0011] Furthermore, S3 specifically refers to: Obtain the control intent triple; Based on the control intent triplet, a device-action feasibility tensor is constructed, where each target device corresponds to a set of acceptable operation types, and a recommended range of control parameters is configured for each operation type. A structural consistency scoring function is constructed by combining the device-action feasibility tensor and the center value of the recommended parameter range to calculate the structural consistency score; The stability score is calculated by combining the gesture trajectory change rate vector and the angle between the user's gaze direction and the target device position vector. The confidence score is obtained by weighted summation based on the structural consistency score and stability score. Based on the confidence score, and combined with the corresponding control intent triplet, an intent quadruple is output.

[0012] Furthermore, the center value of the recommended parameter interval represents the width of the half-interval, which is used to normalize the deviation of the current parameter.

[0013] Furthermore, S4 specifically includes: Obtain the confidence score; If the set confidence level execution threshold is When the confidence score is less than When the system refuses to execute the control action, the user needs to provide further details; when the confidence score is greater than or equal to At that time, the instruction generation process begins; The instruction generation process includes the following: The confidence score is greater than or equal to The intention quadruple is used to introduce a dynamic control gain coefficient into the control parameters, generating adjusted control parameters. The dynamic control gain coefficient is calculated based on the confidence score and the confidence execution threshold. The higher the confidence score is above the confidence execution threshold, the greater the dynamic control gain coefficient tends to be. , To control the maximum increase, indicating that the system trusts the intention, the execution parameters are appropriately amplified for a more agile response; when the confidence score is lower than the confidence execution threshold, no control operation is performed. After completing the amplitude adjustment, the system looks up the device type and control protocol mapping table according to the current target device to determine the communication protocol and control commands of the device; The control command is issued through the device control center. After the command is sent, the device response module waits for the target device to return a confirmation response frame. If no confirmation is received within the set time, the system records the current execution status as failure and prompts the user that there may be a communication error or device failure. If a confirmation is received, the system records the current control behavior and writes the current intent quadruple as a successful behavior into the control log.

[0014] A second aspect of the invention provides a smart home control system based on human-computer interaction, the system comprising: The information acquisition and encoding module is used to acquire the user's voice signal, gesture motion image, head posture information and environmental state information to construct a multimodal interaction sequence; wherein, the multimodal interaction sequence contains several modality fusion vectors at a single moment, and the modality fusion vectors include voice features, gesture features, gaze direction features and environmental state features corresponding to the voice signal, gesture motion image, head posture information and environmental state information; The intent recognition and parsing module is used to input the multimodal interaction sequence into the improved temporal fusion model to identify the user's control intent and output the final fusion representation vector. Based on the final fusion representation vector, a control intent triplet containing the target device, operation type, and control parameters is generated through three sets of parallel discrimination modules. The confidence assessment module is used to evaluate the logical consistency and expression stability of the control intention triples, calculate the confidence score, and combine it with the control intention triples to output the intention quadruples. The instruction generation and execution module is used to convert the corresponding control intent triplet into a control instruction for the corresponding target device and execute it based on the intent quadruple when the corresponding confidence score exceeds a preset threshold. The three sets of parallel discrimination modules also include a spatial location guidance term and a context adversarial regularization term; the spatial location guidance term is used to guide the direction of the final fused representation vector to align with the user's current gaze device; the context adversarial regularization term is used to reduce target selection interference caused by abnormal environmental offsets.

[0015] The beneficial technical effects of the present invention are at least as follows: This invention addresses the shortcomings of existing smart home human-computer interaction systems, such as voice interference, ambiguous gestures, unclear target devices, incomplete commands, and unstable multimodal expressions. It proposes a smart home control method and system based on multimodal interaction understanding and pre-execution confidence assessment. First, this invention constructs a unified, time-aligned multimodal input sequence using four types of inputs: voice, gestures, user gaze direction, and environmental state. This ensures the consistency of semantic cues generated during natural interaction across the temporal dimension. Then, by fusing a temporal modeling structure that integrates voice semantics, gesture trajectory changes, gaze direction, and contextual state, it identifies the user's target device, operation, and desired adjustment range, forming a structured control intent. Furthermore, this invention proposes an intent confidence assessment mechanism for the home environment. By utilizing quantitative indicators such as the structural feasibility relationship between devices and actions, recommended ranges for device parameters, and the consistency between user actions and gaze direction, it assesses the reliability of the identified intent, thereby avoiding erroneous control operations triggered by incomplete voice, insufficient gesture amplitude, or ambiguous expressions. After passing the confidence threshold determination, this invention transforms the structured intent into control protocol instructions for specific devices and introduces a dynamic control amplitude adjustment mechanism based on confidence. This makes the system more agile in response under high confidence conditions and maintains a stable control amplitude under low confidence but executable conditions, further improving the naturalness and security of the control effect. Through a complete link design from input acquisition, intent parsing, confidence judgment to control execution, this invention achieves a natural interactive control mode that is closer to real-world home usage scenarios. It significantly improves the ability to resolve fuzzy expressions, the accuracy of target device determination, and the stability and robustness of the actual execution process, thereby improving the user experience and enhancing the system's usability in complex home environments. Attached Figure Description

[0016] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0017] Figure 1 This is a flowchart of the smart home control method based on human-computer interaction according to the present invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] like Figure 1As shown in the figure, the intelligent home control method based on human-computer interaction provided by the embodiment of the present invention includes: S1. Collect the user's voice signal, gesture motion image, head posture information, and environmental state information to construct a multimodal interaction sequence; wherein, the multimodal interaction sequence contains several modal fusion vectors at a single moment, and the modal fusion vectors include voice features, gesture features, gaze direction features, and environmental state features corresponding to the voice signal, gesture motion image, head posture information, and environmental state information.

[0020] Specifically, this step involves collecting natural user interaction signals in smart home interaction scenarios and performing temporal alignment and structured encoding on information such as voice, gestures, user gaze direction, and environmental state to form a unified input vector sequence, providing a stable input foundation for the subsequent control intent recognition module. This step focuses on addressing the modal differences of the input signals, time synchronization issues, and the construction of information fusion representations, with the goal of obtaining a multimodal interaction state representation expressing the user's current intent at the system level.

[0021] Since this step is the starting point of the process, the input data comes directly from the front-end sensing devices at the smart home system deployment site, mainly including the following four categories: voice signals. User voice is captured via a microphone array, using a layout of 5 or more microphones, with a sampling rate of 16kHz, a signal window length of 25ms, and a frame shift of 10ms; gesture image sequences are also included. Data is captured by an RGB-D camera; a structured light camera with a resolution of 640×480 and a frame rate of 30fps is recommended to record the coordinates of 21 key points per frame; user head pose vector. Facial landmarks are extracted from images from the same camera, and a MobileNet-based facial landmark estimation algorithm is used to output a three-axis direction cosine code; environmental state The illumination is provided by the system's internal time module and light sensor. The time period is identified by the system clock (e.g., divided into morning, noon, and evening). The illumination value is the intensity value after the sensor voltage is converted from analog to digital and normalized.

[0022] Furthermore, to ensure that the input modalities can correctly participate in the subsequent unified modeling process, the system first divides all data into sampling time windows. (Recommended value: 200ms) Uniform segmentation. Each segment corresponds to an interaction time point, and each modal data point is mapped to that time point range, forming a modality-aligned segment. After alignment, feature extraction and encoding operations are performed on each modality. (Speech signal) Mel-ray cepstral features were extracted for each time window and processed using a four-layer convolutional network configured with 64 / 128 / 128 / 128 channels and a kernel size of [missing value]. Output a feature vector with dimension 128, denoted as Gesture sequences The coordinate sequence consists of 21 two-dimensional keypoints. The curvature change of the keypoint trajectory is calculated to form a dynamic description of the gesture. After dimensionality reduction, it becomes a 42-dimensional feature vector, denoted as . Head direction A pose estimation model is constructed using 68 facial key points to output the orientation angle. This is then used to form a three-dimensional orientation vector through cosine mapping, denoted as... Environmental conditions In the process, the time period uses three-dimensional one-heat encoding, the illumination values ​​are normalized to one-dimensional floating-point values, and spliced ​​together to form a two-dimensional environmental feature, denoted as . .

[0023] Finally, the four types of feature vectors are concatenated according to their feature dimensions to form a single-time modality fusion vector: ; in This represents a vector concatenation operation. It consists of 128 dimensions of speech features, 42 dimensions of gesture features, 3 dimensions of head orientation, and 2 dimensions of environmental state. It will continuously... The combination of modal fusion vectors formed over several time periods constitutes a multimodal interaction sequence: ; The sequence This is the output of this step. Each It contains a fused representation of all interaction signals within the time period, possessing temporal consistency, unified information structure, and the low-dimensional representation capability required for subsequent modeling.

[0024] S2. Input the multimodal interaction sequence into the improved temporal fusion model to identify the user's control intention and output the final fusion representation vector; based on the final fusion representation vector, generate a control intention triplet containing the target device, operation type and control parameters through three sets of parallel discrimination modules.

[0025] Specifically, this step is based on the multimodal interactive input sequence constructed in step one. The goal is to achieve structured recognition of users' natural interaction intentions and output a triplet of target device, operation type, and control parameters. Furthermore, the semantic source and decision-making logic of each output should be clearly defined. Since user expressions in smart home scenarios are often highly ambiguous and non-referential, such as "Turn it down for me," "This is a bit noisy," "Turn it down a bit," and coupled with issues such as unstable movement amplitude and missing words in speech segments, conventional speech recognition-semantic extraction chain processing cannot be used. Instead, while modeling temporal information, the collaborative semantics between modalities should be used as the core of modeling, integrating the overall intent structure of the recognition target, action, and parameters.

[0026] Input sequence Modality fusion vector at each time point It already includes four types of information: speech features, gesture features, gaze direction features, and environmental state features. To avoid the inability of ordinary temporal modeling structures to handle interaction scenarios with "fuzzy expressions but clear directions," the system design employs a specially improved temporal fusion model, consisting of a three-layer gated loop structure and a set of modal attention weight matrices. Its output is the final fused representation vector. Building upon the output of the standard GRU, a modal attention residual mechanism is further introduced to dynamically emphasize the contribution of different time points and different modes to the final control intent. This is expressed as: ; in, This represents the encoding result of the input sequence by the standard gated cyclic unit; Indicates the first position in the input sequence Each modality is independently vectored into a subspace (speech, gesture, gaze, environment), with each modality dimension maintaining its original proportions; For the attention weight matrix of each modality, a shallow bilinear structure is used to obtain the weights from the original input. It is obtained through dynamic learning; The modal residual fusion coefficient has a range of values. This is used to control the influence of attention branches on the main path. This structure allows the model to obtain compensatory discrimination criteria through other modalities even when the interaction signal is incomplete, especially demonstrating higher intent stability in scenarios such as "gestures without speech" and "default target in speech".

[0027] Furthermore, fusion represents It is then fed into three parallel discrimination modules to generate the target device. Operation type and control parameters The final structured output. Because device identification numbers have a location-dependent characteristic, meaning they are highly correlated with the user's current gaze direction, and environmental conditions may also interfere with target reasoning (e.g., air conditioning takes precedence over lights at night), spatial location guidance terms and contextual adversarial regularization terms have been added to the device identification module, specifically: ; in, This is the device category discrimination vector. Indicate each device The corresponding spatial gaze reference vector is derived from the installation layout; Indicates inner product operation, used to guide The direction should be aligned with the device the user is currently looking at; The projection vector of the current environment state, with regularization term Used to mitigate target selection interference caused by abnormal environmental offset; , These are two weighting coefficients, controlling the ratio of space guidance to environmental countermeasures. In practical use, The recommended value is 0.3. The recommended value is 0.5. This structure, compared to the traditional softmax classifier, incorporates smart home layout awareness and contextual adversarial mechanisms, representing a structural innovation addressing the issues of fixed distribution of home devices and strong correlation between environmental behaviors. Action Type With parameter value It is generated by the standard feedforward structure. Action label categories (such as "on", "off", "raise", "lower") primarily rely on verb structure in speech and directional gestures. For specific control parameters, the output is quantized according to the actual system control protocol (e.g., brightness level 5, fan speed level 3). In particular, to handle vague expressions such as "turn it down a little" or "slightly brighter," The output is not a fixed value, but a range plus a direction of change, such as "adjust down 1-2 levels from the current value," learned through fuzzy labeling introduced during the training phase. Finally, this step outputs a triplet. , respectively, represent the device objective, action type, and control range that the system should execute in the current interaction.

[0028] S3. Evaluate the logical consistency and expression stability of the control intention triplet, calculate the confidence score, and combine it with the control intention triplet to output the intention quadruple.

[0029] Specifically, this step is used to process the control intent triplet identified in step two. A confidence assessment is performed before execution to determine whether the current intent possesses sufficient logical consistency and expressive stability, thereby deciding whether to allow the system to enter the home device control phase. This step plays a crucial role in this patent system: on the one hand, it directly receives the intent result derived from fusion modal reasoning such as voice, gestures, and gaze direction; on the other hand, it decides whether to convert the result into a real control signal, thus constituting the technical boundary between "understanding" and "execution" of the system. Considering that users in smart home environments often have non-standardized expression habits, such as "It's too cold, adjust it a bit" without specifying a device, or "Turn it down a bit" without giving the parameter strength, this step must design an intent filtering mechanism that can handle "expressive ambiguity" and "structural incompleteness."

[0030] The input is the triplet output from step two. ,in This indicates the identification number of the target device (such as lights, air conditioners, etc.). Indicates the control action label (such as turn on, increase, etc.). This represents the numerical parameter or magnitude level corresponding to the control action. To ensure tolerance for ambiguous expressions while constraining structural conflicts, this step introduces a two-branch confidence modeling structure. The main branch evaluates the semantic consistency of the intent itself, while the auxiliary branch measures the stability and contextual rationality of the user's expression. The two are then combined to form the final confidence score. This will be used for decision-making in the next stage.

[0031] First, the main branch evaluates the structural consistency among the equipment, actions, and parameters.

[0032] To this end, the system pre-constructs a device-action feasibility tensor. Each device Corresponding to a set of acceptable actions and for each action Configure recommended control parameter range These data can be directly extracted from the device protocol definition file. For example, the recommended value range for the "temperature adjustment" operation of an air conditioning unit is... The recommended range for the "brightness adjustment" parameters of lighting equipment is as follows: (Based on brightness level).

[0033] Structural consistency scoring function The definition is as follows: ; in, Score the structural consistency. For indicator functions, when the device Support Actions The value is 1 if the condition is met, and 0 otherwise. The recommended parameter range is the center value, i.e. ; This represents the width of the half-interval, used to normalize the deviation of the current parameter. This is a weighting coefficient used to adjust the importance of structural logic and parameter matching. The score reflects the structural rationality and, by determining the parameter position, whether the control behavior falls near the middle range recommended by the system, thus encouraging users' natural expression behavior to fall within the system's acceptable range.

[0034] Secondly, the auxiliary branch evaluates whether the intent is expressed continuously, clearly, and stably during the interaction. Considering situations where users may "speak completely but gesture intermittently" or "say an action but not specify a device," this branch does not rely on semantic content but calculates a stability score based on the user's interaction behavior patterns. Its expression is as follows: ; in, For stability score, Indicates time The vector of the rate of change of the gesture trajectory at time step. This is a Sigmoid function used to enhance the score of moderate-amplitude movements, while movements that are too strong or too weak will weaken the score. The angle between the user's gaze direction and the target device's position vector. This indicates gaze consistency; the value is close to 1 when the gaze is close to the device and tends to 0 when it is away from the device. This represents the combined weight of the two sub-items.

[0035] The final confidence score is the weighted sum of the two sub-scores mentioned above: ; in The main and auxiliary branch weight adjustment coefficient is suggested to be 0.6, which indicates that more emphasis is placed on the logical consistency of the intention structure itself, but the stability of the modal signal in the expression process is still retained as an auxiliary judgment criterion.

[0036] when Higher than the preset threshold If the current intent is deemed credible, the system can proceed to the next execution stage; otherwise, the system will terminate the current instruction and wait for further expression from the user. The output structure is a quadruple. .

[0037] S4. Based on the intent quadruple, when the corresponding confidence score exceeds a preset threshold, the corresponding control intent triple is converted into a control command for the corresponding target device and executed.

[0038] Specifically, this step is based on the intent quadruple output from step three. This process generates the final home control commands and physically executes them through device control interfaces. Its core task is to transform the action intentions parsed from natural interactions into control signaling formats acceptable to different home devices, while ensuring the credibility of the control intentions. It also considers constraints specific to home scenarios, such as interaction continuity, user command ambiguity, and device response mechanisms, to design an execution mechanism with dynamic adaptability and control security. This step's design fully considers the output structure characteristics of preceding semantic parsing and confidence assessment, and is closely dependent on… The system employs a three-element control structure and introduces a confidence-guided amplitude adjustment mechanism to prevent drastic deviations in control performance caused by slight fluctuations in user feedback.

[0039] Among them, the quadruple This indicates the target device number, for example, number 1 represents the master bedroom light, number 2 represents the living room air conditioner, etc. The number is bound to the control system when the device is registered; The action type label corresponds to a fixed set of instruction semantics, including "on", "off", "raise", "lower", etc. The value represents the control parameter, which is derived from the result of the joint judgment of voice intent and gesture trajectory. The unit varies depending on the device. For example, the temperature adjustment unit of an air conditioner is ℃, and the brightness adjustment unit of a light is a level. The confidence score calculated in step three reflects the clarity and structural consistency of the current control intent. The system's set confidence execution threshold is... ,when When this happens, the system refuses to execute the control action and requires the user to provide further details. At that time, the instruction generation process begins.

[0040] Furthermore, to improve the naturalness of the system's interactive response and avoid excessively large or small control amplitudes, this step introduces a confidence-driven control amplitude adjustment mechanism. Specifically, the system defines a dynamic control gain coefficient. This parameter is used to adjust the parameter range based on the current confidence level, preventing large-scale device jumps in cases of "low confidence but reasonable expression," and also enhancing the system response speed in cases of "high confidence and clear expression." The adjusted control parameter is denoted as... The calculation method is as follows: ; ; in, The adjusted control parameters, As a confidence level adjustment factor, For confidence level, Control and adjust the maximum increase. Control and adjust sensitivity. The function design makes it possible when... Approaching the threshold hour, When the value is close to 1, it indicates that the control amplitude is not adjusted; when Higher than The more, tend This indicates that the system trusts the intent, appropriately amplifies the execution parameters, and responds more nimbly; when If the value is below the threshold, no control operation will be performed.

[0041] Furthermore, after completing the amplitude adjustment, the system adjusts according to the target device. Locate the device type and control protocol mapping table to determine the device's communication protocol (e.g., ZigBee, Wi-Fi, Bluetooth, or infrared) and control format template. The control template includes the control command code corresponding to the action type, parameter field structure, and data type. For example, for device number 2 (assuming it's an air conditioner), if... To "lower," the system looks up a table and determines that a "temperature decrease" command should be generated, with a data type of 1-byte integer and a unit of 1°C; if , ,but The value is rounded down to 2 and written into the instruction field to form a control command frame. The final generated control instruction packet... This includes: device address, command type code, parameter values, verification fields, timestamps, etc., which are encapsulated into a specific communication protocol format and ready to be sent.

[0042] Control commands are issued through the device control center, which is implemented by a local embedded control board (such as an STM32 or ESP32-based SoC) connected to the device via serial port, RF, or infrared module. After transmission, the device response module waits for an acknowledgment frame from the target device. If no acknowledgment is received within a set time, the system records the current execution status as failed and prompts the user that there may be a communication error or device malfunction. If an acknowledgment is received, the system records the current control action and... Successful actions are written into the control log for subsequent user preference learning and behavior modeling.

[0043] For example, if a user says "dimmer" to the living room lights and then swipes down with their finger, the system will interpret the data and obtain... (Living room lighting) = "lower" (Based on brightness level) , ,set up , Substituting into the formula, we get ,but The value is rounded down to 1, generating a command to decrease the brightness by 1 level. This command is sent via the ZigBee protocol. Once the control module receives an acknowledgment after sending the command, the control operation is considered successful, and the command and confidence level are recorded in the log.

[0044] This invention also provides a smart home control system based on human-computer interaction, the system comprising: The information acquisition and encoding module is used to acquire the user's voice signal, gesture motion image, head posture information and environmental state information to construct a multimodal interaction sequence; wherein, the multimodal interaction sequence contains several modality fusion vectors at a single moment, and the modality fusion vectors include voice features, gesture features, gaze direction features and environmental state features corresponding to the voice signal, gesture motion image, head posture information and environmental state information; The intent recognition and parsing module is used to input the multimodal interaction sequence into the improved temporal fusion model to identify the user's control intent and output the final fusion representation vector. Based on the final fusion representation vector, a control intent triplet containing the target device, operation type, and control parameters is generated through three sets of parallel discrimination modules. The confidence assessment module is used to evaluate the logical consistency and expression stability of the control intention triples, calculate the confidence score, and combine it with the control intention triples to output the intention quadruples. The instruction generation and execution module is used to convert the corresponding control intent triplet into a control instruction for the corresponding target device and execute it based on the intent quadruple when the corresponding confidence score exceeds a preset threshold. The three sets of parallel discrimination modules also include a spatial location guidance term and a context adversarial regularization term; the spatial location guidance term is used to guide the direction of the final fused representation vector to align with the user's current gaze device; the context adversarial regularization term is used to reduce target selection interference caused by abnormal environmental offsets.

[0045] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0046] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0047] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A smart home control method based on human-computer interaction, characterized in that, The method includes: S1. Collect the user's voice signal, gesture motion image, head posture information and environmental state information to construct a multimodal interaction sequence; wherein, the multimodal interaction sequence contains several modality fusion vectors at a single moment, and the modality fusion vectors include voice features, gesture features, gaze direction features and environmental state features corresponding to the voice signal, gesture motion image, head posture information and environmental state information; S2. Input the multimodal interaction sequence into the improved temporal fusion model to identify the user's control intention and output the final fusion representation vector; based on the final fusion representation vector, generate a control intention triplet containing the target device, operation type and control parameters through three sets of parallel discrimination modules; S3. Evaluate the logical consistency and expression stability of the control intention triplet, calculate the confidence score, and combine it with the control intention triplet to output the intention quadruple. S4. Based on the intent quadruple, when the corresponding confidence score exceeds a preset threshold, the corresponding control intent triple is converted into a control command for the corresponding target device and executed. The three sets of parallel discrimination modules also include a spatial location guidance term and a context adversarial regularization term; the spatial location guidance term is used to guide the direction of the final fused representation vector to align with the user's current gaze device; the context adversarial regularization term is used to reduce target selection interference caused by abnormal environmental offsets.

2. The smart home control method based on human-computer interaction according to claim 1, characterized in that, The voice signal is acquired through a microphone array; the gesture image is acquired by an RGB-D camera; the head pose information is extracted from facial key points in the same camera image, and a MobileNet-based facial key point estimation algorithm is used to output a three-axis direction cosine code. The environmental status information is provided by the system's internal time module and light sensor. The time period is identified by the system clock, and the light value is the intensity value of the sensor voltage after analog-to-digital conversion and normalization.

3. The smart home control method based on human-computer interaction according to claim 1, characterized in that, The multimodal interaction sequence is generated as follows: The voice signal, gesture image, head posture information, and environmental state information are uniformly segmented according to the sampling time window; each segment corresponds to an interaction moment, and each modal data is mapped to the range of that moment to form a modality-aligned segment; After alignment is completed, feature extraction and encoding operations are performed on the modal alignment fragments of each modality to obtain the corresponding feature vectors; The corresponding feature vectors are concatenated according to the feature dimension to form a modality fusion vector at a single moment; The modal fusion vectors from consecutive time periods are combined to form a multimodal interaction sequence.

4. The smart home control method based on human-computer interaction according to any one of claims 2 or 3, characterized in that, After alignment, feature extraction and encoding operations are performed on the aligned segments of each modality to obtain the corresponding feature vectors, specifically: Mel-spectral signatures were extracted from the speech signal at each time window and processed using a four-layer convolutional network configured with 64 / 128 / 128 / 128 channels and a kernel size of [missing value]. Output a feature vector with a dimension of 128, which is the speech feature; The gesture motion image consists of a coordinate sequence of 21 two-dimensional key points. The curvature change of the key point trajectory is calculated to form a dynamic description of the gesture. After dimensionality reduction, it becomes a 42-dimensional feature vector, which is the gesture feature. The head pose information is used to construct a pose estimation model using 68 facial key points to output the direction angle, and then a three-dimensional direction vector is formed through cosine mapping, which is the gaze direction feature; In the environmental state information, the time period is encoded using three-dimensional one-hot encoding, the illumination value is normalized to a one-dimensional floating-point value, and the two-dimensional environmental features are spliced ​​together to form the environmental state features.

5. The smart home control method based on human-computer interaction according to claim 1, characterized in that, The improved temporal fusion model consists of a three-layer gated loop structure and a set of modal attention weight matrices, and its output is the final fusion representation vector; wherein, the modal attention weight matrix is ​​used to dynamically emphasize the contribution of different time points and different modalities to the final control intention.

6. The smart home control method based on human-computer interaction according to claim 1, characterized in that, The spatial location guidance term is obtained by performing an inner product operation on the final fused representation vector and the spatial gaze reference vector corresponding to each target device; the context adversarial regularization term is calculated by combining the final fused representation vector and the current environment state projection vector.

7. The smart home control method based on human-computer interaction according to claim 1, characterized in that, Specifically, S3 is: Obtain the control intent triple; Based on the control intent triplet, a device-action feasibility tensor is constructed, where each target device corresponds to a set of acceptable operation types, and a recommended range of control parameters is configured for each operation type. A structural consistency scoring function is constructed by combining the device-action feasibility tensor and the center value of the recommended parameter range to calculate the structural consistency score; The stability score is calculated by combining the gesture trajectory change rate vector and the angle between the user's gaze direction and the target device position vector. The confidence score is obtained by weighted summation based on the structural consistency score and stability score. Based on the confidence score, and combined with the corresponding control intent triplet, an intent quadruple is output.

8. The smart home control method based on human-computer interaction according to claim 7, characterized in that, The center value of the recommended parameter interval represents the width of the half-interval, which is used to normalize the deviation of the current parameter.

9. The smart home control method based on human-computer interaction according to claim 1, characterized in that, Specifically, S4 is: Obtain the confidence score; If the set confidence level execution threshold is When the confidence score is less than When the system refuses to execute the control action, the user needs to provide further details; when the confidence score is greater than or equal to At that time, the instruction generation process begins; The instruction generation process includes the following: The confidence score is greater than or equal to The intention quadruple is used to introduce a dynamic control gain coefficient into the control parameters, generating adjusted control parameters. The dynamic control gain coefficient is calculated based on the confidence score and the confidence execution threshold. The higher the confidence score is above the confidence execution threshold, the greater the dynamic control gain coefficient tends to be. , To control the maximum increase, indicating that the system trusts the intention, the execution parameters are appropriately amplified for a more agile response; when the confidence score is lower than the confidence execution threshold, no control operation is performed. After completing the amplitude adjustment, the system looks up the device type and control protocol mapping table according to the current target device to determine the communication protocol and control commands of the device; The control command is issued through the device control center. After the command is sent, the device response module waits for the target device to return a confirmation response frame. If no confirmation is received within the set time, the system records the current execution status as failure and prompts the user that there may be a communication error or device failure. If a confirmation is received, the system records the current control behavior and writes the current intent quadruple as a successful behavior into the control log.

10. A smart home control system based on human-computer interaction, characterized in that, The system includes: The information acquisition and encoding module is used to acquire the user's voice signal, gesture motion image, head posture information and environmental state information to construct a multimodal interaction sequence; wherein, the multimodal interaction sequence contains several modality fusion vectors at a single moment, and the modality fusion vectors include voice features, gesture features, gaze direction features and environmental state features corresponding to the voice signal, gesture motion image, head posture information and environmental state information; The intent recognition and parsing module is used to input the multimodal interaction sequence into the improved temporal fusion model to identify the user's control intent and output the final fusion representation vector. Based on the final fusion representation vector, a control intent triplet containing the target device, operation type, and control parameters is generated through three sets of parallel discrimination modules. The confidence assessment module is used to evaluate the logical consistency and expression stability of the control intention triples, calculate the confidence score, and combine it with the control intention triples to output the intention quadruples. The instruction generation and execution module is used to convert the corresponding control intent triplet into a control instruction for the corresponding target device and execute it based on the intent quadruple when the corresponding confidence score exceeds a preset threshold. The three sets of parallel discrimination modules also include a spatial location guidance term and a context adversarial regularization term; the spatial location guidance term is used to guide the direction of the final fused representation vector to align with the user's current gaze device; the context adversarial regularization term is used to reduce target selection interference caused by abnormal environmental offsets.

Images