Smart home control method and system based on multiple modes

Abstract

The invention relates to the technical field of intelligent control, in particular to a multi-mode-based intelligent home control method and system, and the method comprises the following steps: S1, synchronously collecting a voice stream and an action video stream of a user, and respectively extracting a semantic anchor point of a voice instruction and a spatial anchor point of an action track; s2, constructing a virtual time axis, mapping the semantic anchor points and the spatial anchor points to a unified space-time coordinate system, and performing anchor point deviation correction on a mapping result based on an environment semantic field to generate an alignment instruction set; and S3, performing intention fusion on the alignment instruction set according to the spatial topological relation of the equipment, generating an equipment control instruction, and triggering execution. According to the method, the fault tolerance and execution rationality of fuzzy instructions or wrong finger behaviors are improved, the method is particularly suitable for family scenes with dense multiple devices and frequent environment dynamic changes, and the practicability and the intelligent level of the system are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control, and particularly relates to a multi-modal based intelligent home control method and system. BACKGROUND

[0002] With the wide deployment of intelligent home systems, users have higher requirements for the naturalness and accuracy of human-computer interaction. In recent years, multi-modal interaction has become an important direction for improving the understanding ability of intelligent control systems. Some systems attempt to jointly analyze voice and images to achieve comprehensive recognition of user "saying" and "pointing". However, existing multi-modal fusion technologies mostly stop at shallow synchronization or static mapping of modal data, lacking systematic modeling of factors such as "time sequence", "spatial direction", and "environment adaptation". Especially when users use ambiguous instructions such as "turn on this" and "dim that", existing methods often cannot accurately understand their true intentions, leading to unclear or false triggering of control devices.

[0003] In addition, current intelligent home systems generally use device control logic based on fixed rules, lacking a deep fusion mechanism for user behavior habits, environmental state, and topological relationships between devices, making it difficult to achieve stable and reliable linkage control in a home environment with multiple devices and complex control intentions.

[0004] In summary, the existing technology still lacks a multi-modal intelligent home control method that takes into account semantic understanding, spatial direction, environmental state adaptation, and device collaborative execution capability. SUMMARY

[0005] The present application provides a multi-modal based intelligent home control method and system, which innovatively designs in aspects such as multi-modal alignment mechanism, semantic field correction mechanism, and topological fusion control strategy, to achieve a more natural, accurate, and intelligent home control experience.

[0006] A multi-modal based intelligent home control method, comprising the following steps:

[0007] S1, synchronously collecting voice stream and action video stream of a user, and extracting semantic anchor points of voice instructions and spatial anchor points of action trajectories, respectively;

[0008] S2, constructing a virtual time axis, mapping the semantic anchor points and spatial anchor points to a unified space-time coordinate system, and based on an environmental semantic field, correcting the anchor points, and generating an alignment instruction set;

[0009] S3, according to device spatial topological relationship, fusing the alignment instruction set, generating a device control instruction, and triggering execution.

[0010] Optionally, the S1 comprises collecting a voice stream and a motion video stream by a time-stamped multi-modal sensor array, the multi-modal sensor array comprising:

[0011] A distributed microphone group records the voice stream;

[0012] A binocular depth camera captures the motion video stream and generates coordinates of hand key points.

[0013] Optionally, the extracting semantic anchors of voice instructions comprises:

[0014] Performing endpoint detection on the voice stream to segment valid instruction segments;

[0015] Identifying a combination of an operation verb and a target pronoun in the instruction segment as a semantic anchor;

[0016] The extracting spatial anchors of motion trajectories comprises:

[0017] Tracking a hand key point motion trajectory and calculating a moving speed change rate;

[0018] When the speed change rate is lower than a speed change rate threshold, taking coordinates of the current frame hand key point as a spatial anchor.

[0019] Optionally, the constructing a virtual time axis comprises taking a time point when the semantic anchor appears as a reference time point, extending a certain period of time forward and backward respectively, and constructing a semantic-driven time window:

[0020] In the forward extension, the time length is dynamically adjusted according to the number of words of the voice instruction, and the more complex the instruction is, the longer the reserved time is;

[0021] In the backward extension, the time range is adjusted according to the historical average motion response delay of the user and the fluctuation degree thereof.

[0022] Optionally, the mapping the semantic anchor and the spatial anchor to a unified space-time coordinate system specifically comprises extracting all hand key points determined as spatial anchors within the time window, and mapping them from a local coordinate system of the camera to a global space coordinate system with a home center as an origin, the mapping process combining a spatial offset relationship between the camera and the home center and a coordinate proportion conversion parameter, all mapped points will be attached with original time stamps, forming a sequence of spatial anchors with time labels.

[0023] Optionally, the anchor point rectification of the mapping result based on the environmental semantic field comprises verifying the sequence of spatial anchors based on a real-time environmental semantic field, and when a spatial anchor pointing position conflicts with an environmental semantic field state is detected, performing a rectification operation:

[0024] (i) If the semantic anchor point includes device operation instructions and points to a non-compatible state of the device, search for the nearest compatible device along the device topology network;

[0025] (ii) If the semantic anchor point includes environment adjustment instructions and points to a region environment parameter exceeding the device calibration threshold, shift the spatial anchor point to the gradient optimization direction of the semantic field.

[0026] Optionally, the generation of the alignment instruction set includes associating the spatial anchor point after the rectification operation with the semantic anchor point by timestamp, and generating a structured instruction tuple: <action type, target position, timestamp>.

[0027] Optionally, the S3 specifically includes:

[0028] S31, for each instruction tuple in the alignment instruction set, calculate the spatial correlation degree score of its target position with each smart device;

[0029] S32, filter the devices with a spatial correlation degree score greater than a score threshold as a candidate device set, and verify the compatibility of the current state of the candidate devices in the candidate device set with the instruction action type;

[0030] S33, intent fusion decision: when a single instruction is associated with multiple compatible devices, execute a fusion strategy:

[0031] (i) If the instruction is an environment adjustment type operation, select the device with the highest spatial correlation degree to execute;

[0032] (ii) If the instruction is a scene triggering type operation, activate all associated devices in the topology network to form a device group for cooperative execution;

[0033] S34, according to the device response delay prediction value, interpolate and sort the instruction execution time sequence in the device group, encode the final instruction sequence into a device executable protocol message and trigger execution.

[0034] Optionally, the spatial correlation degree score is calculated based on the Euclidean distance of the device and the target position, the projection weight of the device in the user's visual field direction, and the historical operation frequency weight.

[0035] A multi-modal based smart home control system for implementing the multi-modal based smart home control method described above, comprising the following modules:

[0036] Voice acquisition and analysis module: for acquiring user voice stream and extracting semantic anchor points;

[0037] Action perception and recognition module: for acquiring user action video stream and extracting spatial anchor points;

[0038] The space-time alignment and rectification module constructs a virtual time axis taking the semantic anchor point as a reference, maps the semantic anchor point and the space anchor point to a unified space-time coordinate system, rectifies the anchor points based on an environment semantic field, and generates an alignment instruction set;

[0039] The intention fusion and control instruction generation module fuses the intention of the alignment instruction set based on the device space topology relationship, and outputs a control instruction for driving the smart device.

[0040] The present application has the following advantages:

[0041] The present application constructs a virtual time axis taking the semantic anchor point as a reference, accurately captures the space anchor point related to the voice instruction in the time domain, and projects it to the global coordinate system of the home through a unified coordinate mapping mechanism, solving the alignment failure problem caused by factors such as time sequence error and inconsistent coordinate systems in multi-modal input. In particular, adjustable time window length and coordinate scale conversion parameters are used, taking into account the diversity of user language style, instruction complexity and camera deployment method, so that the system can respond to ambiguous expressions such as "this" and "there" in a natural context, greatly improving the intuitiveness of the interaction and user experience.

[0042] The present application proposes an environment semantic field rectification mechanism to address the inconsistency between user semantic intention and physical environment state (e.g., "turn on the light" pointing to a light that is already on, "dim" pointing to a strong light area), which first maps real-time perceived device state, light intensity and other parameters into a semantically recognizable field structure, and dynamically corrects the pointing position of the space anchor point in combination with the device topology network or the light gradient direction, achieving rectification capability based on semantic consistency. This mechanism improves the fault tolerance and execution rationality of ambiguous instructions or misdirected behavior, and is particularly suitable for multi-device dense and frequently changing environment scenarios, significantly enhancing the practicality and intelligence level of the system.

[0043] The present application designs a space topology correlation degree calculation model, which takes into account the spatial distance between the target position and the device, the user's field of view weight and the historical operation frequency in the weighted score, and establishes a device matching mechanism with "space + semantics + habits" three-dimensional weight. This mechanism not only supports optimal device selection for adjustment type instructions, but also cooperates with device group expansion logic and device timing optimization strategy for scene triggered instructions, which can automatically determine the execution order, delay interpolation and protection interval in a multi-device linkage scene, thereby realizing the precision and intelligence of collaborative control among multiple devices, and is particularly suitable for efficient response in cross-device scenarios such as cinema mode and sleep mode. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to make the technical solutions in the present application or prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only aim to explain the present application, and for those skilled in the art, other drawings can also be obtained without creative effort.

[0045] Figure 1 The control method flowchart of the embodiment of the present application is shown in the figure.

[0046] Figure 2 The control system function module schematic diagram of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0047] The present application will be described in detail below in combination with the accompanying drawings and specific embodiments. For some known technologies, other alternative ways can also be implemented by those skilled in the art; and the accompanying drawings are only used to more specifically describe the embodiments, and are not intended to specifically limit the present application.

[0048] As shown in the figure, a multi-modal based smart home control method includes the following steps: Figure 1

[0049] S1, synchronously collect the user's voice stream and action video stream, and extract the semantic anchor point of the voice instruction and the spatial anchor point of the action trajectory, respectively.

[0050] The semantic anchor point refers to the key semantic combination with operation meaning in the voice instruction, which is composed of an operation verb (such as "open" and "close") and a target pronoun (such as "this" and "there"), and is used to identify the user's control intention and target object. In the present application, the extraction process of the semantic anchor point includes:

[0051] Endpoint detection: first, the energy threshold judgment and the silent section exclusion are performed on the original voice stream, and the voice segment containing the actual instruction content is extracted;

[0052] Key word recognition: in the valid voice segment, it is checked whether the predefined operation verb and target pronoun appear at the same time, and if so, they are combined into a semantic anchor point.

[0053] The spatial anchor point refers to the position point in the user's action trajectory that can reflect the intention direction or locate the specific moment of the intention, which occurs when the gesture slows down or stops, and represents the user's attention or direction to a certain spatial position. In the present application, the extraction process of the spatial anchor point includes:

[0054] Trajectory tracking: the three-dimensional position of the user's hand is recognized and tracked through a depth camera to form a continuous coordinate sequence;

[0055] ​Velocity change rate calculation: Calculate the velocity change rate (i.e. acceleration) for each frame of hand position and compare it to a pre-set threshold;

[0056] Anchor point determination: When the velocity change rate of a frame is lower than the threshold (indicating that the hand gesture has significantly decelerated or paused), the three-dimensional coordinates corresponding to that frame are taken as a spatial anchor point.

[0057] S1 specifically includes the following steps:

[0058] S11, collect user input data through a time-stamped multi-modal sensor array, which includes:

[0059] Distributed microphone group: record voice stream in real time at a sampling rate f s = 16 kHz, improving far-field voice capture capability; the main frequency components of human voice are concentrated between 300 Hz and 3400 Hz, to ensure complete capture of voice information, the Nyquist sampling theorem must be met, i.e. the sampling rate should be greater than twice the highest frequency of the voice. 16 kHz can cover a frequency range of up to 8 kHz, which meets the needs of voice recognition and also considers signal-to-noise ratio and storage / computing overhead in a far-field environment, and is a common balanced choice in current intelligent voice processing (such as VAD, ASR).

[0060] Dual-eye depth camera: capture motion video stream at a frame rate f v = 30 fps, and output a sequence of three-dimensional coordinates of hand key points: represents the coordinates of the hand key points in the t-th frame in three-dimensional space, x t , y t , z t are the coordinate values of the hand in the X (horizontal), Y (vertical), and Z (depth) directions, respectively, with units usually in meters (m) or centimeters (cm), depending on the calibration method of the depth camera; motion trajectory recognition, especially hand movement tracking, requires sufficient time resolution to ensure trajectory continuity and the accuracy of anchor point extraction. 30 fps (30 frames per second) is sufficient to cover most of the motion speed changes of daily gestures, and can better restore dynamic features such as hand acceleration.

[0061] Synchronization mechanism: integrate a hardware-level timestamp synchronizer to ensure that the time alignment of multi-modal data meets: |Δt audio-video | < 5 ms; where Δt audio-video is the time difference between the voice frame and the video frame, meeting the IEEE 1857-2023 multi-modal timing synchronization standard.

[0062] S12, semantic anchor point extraction of voice instructions:

[0063] S121, endpoint detection (VAD): perform energy threshold detection on the original voice signal to extract valid voice segments: E represents the energy of the current frame frame greater than a set energy threshold θ energy and not belonging to a non-silence region, it is determined that the frame belongs to a valid speech segment. This is the basic judgment logic of voice activity detection (VAD). Among them, E frame represents the energy of the current frame, non-silence represents a non-silence frame (confirmed by the noise determinator), θ energy represents the energy threshold (empirical setting), the value range is 40-60dB (in the logarithmic scale of short-time energy), and the specific value is dynamically adjusted according to the ambient noise level. If in a relatively quiet environment, generally set to 45dB can better distinguish speech and silence, valid segment represents a speech segment containing valid instructions.

[0064] S122, semantic anchor point recognition: search for semantic structure in the valid speech segment, and identify combined keywords:

[0065] In the extracted valid speech segment, if an operation verb v and a target pronoun p are detected at the same time, they are combined to form a semantic anchor A s , which is used for subsequent instruction intention recognition and modal alignment.

[0066] Among them:

[0067] A s represents a semantic anchor point;

[0068]

[0069]

[0070] S13, spatial anchor point extraction of action trajectory:

[0071] S131, trajectory tracking: use the skeleton key point output to track the hand key point coordinate sequence:

[0072] S132, speed change rate calculation: the speed change rate (acceleration) is obtained by taking the derivative of the hand motion speed between consecutive frames:

[0073] Among them

[0074] represents the hand speed of the t-th frame, represents the hand speed of the t-1 frame, at represents the speed change rate (acceleration) of the t-th frame, and Δt represents the inter-frame time interval,

[0075] S133, spatial anchor point determination rule: when the following conditions are met, the hand three-dimensional coordinates of the t-th frame are determined as spatial anchor points: A p represents a set of spatial anchor points, and θ a is a speed change rate threshold value for determining a stationary state and extracting spatial anchor points:

[0076]

[0077] S2, constructing a virtual time axis, mapping the semantic anchor points and the spatial anchor points to a unified space-time coordinate system, and based on the environment semantic field, correcting the mapping results to generate an alignment instruction set.

[0078] S2 specifically includes the following steps:

[0079] S21, virtual time axis construction: taking the appearance time of the semantic anchor point as the reference time point T0, a dynamically expanding time window is constructed: T ∈ [T0- δt1, T0+ δt2]; wherein T0 represents the reference time point of the appearance of the semantic anchor point, δt1 represents the time length of the forward expansion of the time window (determined according to the length of the voice instruction), δt1 = k × N w wherein k is a time constant corresponding to each word, k = 0.3 s / word, N w is the number of words of the voice instruction in which the semantic anchor point is located; δt2 represents the time length of the backward expansion of the time window (determined according to the user response delay), δt2 = μ delay + 2σ, wherein μ delay is the average historical action response delay of the user, and σ is the fluctuation standard deviation thereof.

[0080] Dynamic adjustment rule: if the voice contains a complex orientation word (such as "the second one on the left"), an additional 0.5 s is added, and for elderly users, an additional 0.8 s is added by default.

[0081] In the present application, the virtual time axis is a time reference framework for aligning semantic anchor points and spatial anchor points. First, the time point when the semantic anchor point (i.e., the key action word and pronoun combination in the user voice instruction) appears is taken as the reference time point, denoted as T0, and then a certain time range is extended forward and backward from T0, respectively, to form a complete time window for screening the hand motion trajectory collected in the time period. The forward extension length (i.e., the reserved time) of the time window is dynamically adjusted according to the complexity of the voice instruction, and the longer the instruction, the more the reserved time. The backward extension length (i.e., the response time required by the user to complete the action) is adjusted individually by referring to the historical response behavior of the user and considering the volatility of the action delay.

[0082] In addition, in order to adapt to different semantic expressions and user groups, the system also designs dynamic adjustment rules, for example:

[0083] When the instruction contains a complex orientation description (such as "the second one on the left side"), the forward time is automatically increased;

[0084] For elderly users, the backward time is extended by default to ensure the fault tolerance and integrity of the system response.

[0085] S22, space-time coordinate mapping: mapping the spatial anchor points in the time window to a three-dimensional global coordinate system with the home center as the origin, forming a space anchor point sequence with a time stamp:

[0086] It represents that in the time window [T0-δt1, T0+δt2] centered on the appearance time T0 of the semantic anchor point, all spatial anchor points after coordinate conversion are extracted, and a time stamp is added to each anchor point to form a space anchor point sequence with a time label

[0087] Coordinate conversion method:

[0088]

[0089] Where (x t ,y t ,z t ) represents the three-dimensional position of the hand in the camera coordinate system, (x c ,y c ,z c ) represents the origin coordinate of the camera, (x o ,y o ,z o ) represents the home center coordinate (global coordinate origin), k1, k2, k3 represent the coordinate scale conversion coefficients based on device calibration, t represents the time stamp of the corresponding spatial anchor point, and x'(t), y'(t), z'(t) represent the three-dimensional coordinates of the mapped spatial anchor point in the home coordinate system.

[0090] If the depth camera device has its own intrinsic calibration parameters, the depth mapping ratio of each axis provided by the manufacturer can be used as the calibration value. If the manufacturer does not calibrate, the calibration process of the coordinate scale conversion coefficients k1, k2, and k3 adopts the device calibration board method: a calibration board with known size and three-dimensional marker points is placed within the visual range of the camera, and by comparing the distance between the points collected by the camera with the actual physical size, the pixel-actual distance ratio in the X, Y, and Z directions is calculated, i.e., the scaling coefficients of the three axes are obtained.

[0091] In multi-modal smart home control, users often issue instructions through voice and gestures together, such as "turn on this" and point to a certain area. However, due to the complexity of the home environment, user expressions may be ambiguous, ambiguous, or conflicting, for example:

[0092] The user points to a lamp that is already on and says "turn on this";

[0093] The user says "dim", but points to a very bright sunlight area;

[0094] The user's expression deviates from the environment device state or physical environment.

[0095] Traditional voice-action fusion schemes mostly rely on user expressions as the only basis, lacking the ability to respond to the current environment state, resulting in inaccurate or ineffective instruction execution. Therefore, a mechanism that can identify, analyze, and automatically correct semantic and environmental conflicts, i.e., "environmental semantic field correction", is needed.

[0096] The core idea of the environmental semantic field correction mechanism is to further introduce real-time sensing capabilities of device states and environmental parameters in the home environment based on the preliminary alignment relationship between semantic anchors (expressing user intent) and spatial anchors (expressing user pointing), forming a "semantic field" to determine whether the anchor point is pointing to the current state logically consistent. Once a contradiction is detected, the target location is dynamically corrected to ensure that the final generated control instruction has reasonable execution and environmental consistency.

[0097] The correction mechanism is divided into two main cases:

[0098] (1) Device state conflict processing, judgment logic: when the semantic anchor intent is "turn on", but the device pointed to by the spatial anchor is currently in the "on" state, there is a logical conflict between the action semantics and the device state.

[0099] Processing strategy: search along the home device topology network for the closest device of the same type but in the "off" state from the original anchor point location;

[0100] Optimization direction: preferentially select the device facing the user direction, or select the commonly used device according to the user behavior portrait, and improve the semantic fit degree of the alternative target;

[0101] Automatically migrate user instruction targets to avoid misoperation or invalid response.

[0102] (2) Environmental parameter conflict processing, judgment logic: when the semantic anchor point is expressed as an environmental regulation instruction (such as "dim" "cooling"), but the physical parameter (such as light intensity) of the region where the spatial anchor point is located is much higher than the set threshold, it indicates that the user points to the region is not suitable for the current instruction;

[0103] Processing strategy: based on the light distribution map obtained by the sensor, calculate the light gradient direction of the current region, that is, find the "decreasing light direction";

[0104] Offset operation: take the original anchor point as the starting point, and displace along the gradient direction with small step, automatically adjust the target position, and finally anchor to the region more suitable for executing the regulation action;

[0105] The instruction automatically points to the region more suitable for the semantic intention, improves the intelligent matching degree and execution efficiency of the control instruction.

[0106] Introduce the concept of "semantic field", convert the environmental state (such as device switch state, light intensity) into a logically readable semantic space, realize the fusion reasoning of semantics and perception, so that the control instruction not only "understand, understand", but also "judge whether it is reasonable". Improve the fault tolerance and adaptability of smart home control, avoid execution failure or misoperation, support fuzzy instruction understanding and correction, adapt to non-accurate expression of natural language and natural action, build a semantic atlas, and provide an interpretable semantic basis for subsequent learning intelligent agents (such as active scene recommendation).

[0107] Conflict detection mechanism: based on the following logical judgment whether the semantic anchor point and the spatial anchor point exist environmental semantic conflict:

[0108] If the semantic anchor point action type ∈ {on, off, start, stop}, check whether the current state of the target device pointed by the spatial anchor point is logically conflicted with the action;

[0109] If the semantic anchor point is a regulation instruction (such as "dim" "cooling" "heating"), it is judged whether the environmental parameter of the pointed region has reached or exceeded the threshold value of the adjustable range.

[0110] The result of executing conflict detection is a Boolean value, which is used to trigger the subsequent deviation correction module.

[0111] S23, environmental semantic field deviation correction mechanism:

[0112] S231, case 1 - device state conflict: for the spatial anchor point sequence Perform semantic consistency check, if the pointing position conflicts with the semantic state of the environment, start the corresponding rectification strategy If the semantic anchor point action is "turn on", and the current state of the device pointed by the spatial anchor point is "turned on", perform alternative target search, and the rectification strategy is:

[0113] Search for a candidate device set that meets the following conditions:

[0114] 1. Match the semantic anchor point action type (such as a lamp);

[0115] 2. Currently in a non-target state (such as "off");

[0116] 3. Located within a certain radius of the original anchor point;

[0117] Select the target device according to the priority:

[0118] 1. The device closest to the user's line of sight / orientation;

[0119] 2. The device with the highest usage frequency in the past 7 days;

[0120] 3. Devices in the current space (such as the same room) are preferred over other spaces.

[0121]

[0122] Conditions to be met:

[0123]

[0124] Priority is given to user line of sight orientation devices and high-frequency use devices, where, represents the three-dimensional coordinates of the current space anchor point, i.e. the pointing position of the user, represents the three-dimensional coordinates of a certain candidate device in space, represents the topological network set of devices in the home (i.e. all candidate control devices), represents the device and the spatial distance between the current anchor point , type represents the type attribute of the device (such as lamps, air conditioners, etc.), state represents the current working state of the device (such as on / off).

[0125] Standard Euclidean distance is used for spatial positioning:

[0126] represents the position coordinates of the candidate device, Represent original space anchor coordinates (user pointing), sorted by distance from near to far, select the closest one in the device that meets the state condition (such as "off").

[0127] Define device topology network structure The network contains the following structure information:

[0128] Each device node Includes: physical coordinates (x, y, z), device type (light, air conditioner, curtain, etc.), current state (on, off, adjusting, etc.), logical label (room to which it belongs, device group ID, etc.), and semantic preference (based on user historical operation frequency, orientation, etc.).

[0129] Device topology network construction process:

[0130] When deployed, build an initial space map by panoramic scanning;

[0131] Linkage smart home protocol (such as Zigbee, Z-Wave, Matter) automatically registers connected devices;

[0132] The background mounts each device to the logical node in the space topology graph through home layout modeling;

[0133] Periodically update the semantic preference of each device node according to user operation records.

[0134] S232, Case 2 - Environment parameter conflict: If the semantic anchor intention involves adjusting the environment state, such as "dim", and the current anchor area physical environment parameter does not meet the intention, such as the pointing area light intensity exceeding the upper limit (such as >1000 lux), then based on the semantic field parameter, i.e. The light gradient direction shifts the anchor to a more suitable position:

[0135] S2321, Get the environment gradient direction (take light as an example): Real-time perception of the environment parameter field of the whole space, such as light distribution I(x, y, z), and get the gradient direction by the following method:

[0136] Perform spatial difference on the local 3x3x3x3 area at the anchor point to estimate the environment gradient vector:

[0137] X direction: compare the average light values of the left and right neighborhoods;

[0138] Y direction: compare the average light values of the upper and lower neighborhoods;

[0139] Z direction: compare the average light values of the front and rear (near and far) neighborhoods.

[0140] Get unit vector Indicates the direction in which the environment parameter decreases the fastest;

[0141] Other extensible semantic field variables include: temperature distribution field T(x, y, z), noise field N(x, y, z), humidity distribution H(x, y, z);

[0142] S2322, gradient offset operation: translate the original anchor point coordinate A small distance along the gradient direction to get a new target point, the step length is set to 0.5m by default, which can be adjusted as needed. If the new point still does not meet the conditions, the offset can be iterated or the user can be prompted that the fuzzy instruction fails. The operation finally obtains the corrected anchor point coordinate For instruction generation.

[0143] Specifically represented as: Wherein, The original space anchor point coordinate (before correction) is represented by The corrected space anchor point coordinate (used to replace the original position) is represented by The light gradient vector pointing to the direction of reduced light is represented by s=0.5m, which represents the offset step (adjustable), and the corrected space anchor point For subsequent instruction fusion.

[0144] S24, generate alignment instruction set: pair the corrected space anchor point with the semantic anchor point according to the timestamp to generate a structured instruction tuple: I = <action type, target position Timestamp t>.

[0145] Table 1 structured instruction tuple example table

[0146]

[0147] S3, according to the device space topology relationship, the alignment instruction set is fused, the device control instruction is generated and executed.

[0148] S3 specifically includes the following steps:

[0149] S31, topology correlation calculation: for each alignment instruction tuple Match its target position With all home devices in space, calculate the spatial topology correlation score S, and the calculation model is as follows:

[0150]

[0151] Wherein, S is the spatial correlation score (the higher the score, the more relevant), du is the Euclidean distance between the target position And the device position, W viewThe projection area ratio of the device in the user's visual field cone, i.e. the projection weight (0~1), F represents the frequency of the device being operated in the past 7 days (normalized), and a=0.6, b=0.3, g=0.1 represent weight coefficients that can be dynamically adjusted by training. Compare the score S of each device with the set threshold value t to screen out the candidate device set that meets the condition The value range of t is 0.3~0.5.

[0152] The spatial topology correlation score S is used to comprehensively evaluate the spatial and semantic correlation strength between the user instruction target position and each candidate device, so as to accurately select the device.

[0153] 1. Distance inverse factor The spatial distance between the device and the user pointing position is a basic factor for judging relevance. The user's gesture usually has clear spatial directionality, so the closer the device, the more likely it will be selected. By taking the inverse form (i.e. the smaller the distance, the higher the score), this item reflects the intuitive logic of physical proximity priority.

[0154] 2. Visual projection weight W view : Considering that the user does not always face the target when giving instructions, the system introduces a user visual field model (such as a visual field cone) to identify whether the device is in the user's visual attention area. If the device is "seen" in the user's visual field, its score should be appropriately increased, and this item reflects the implicit intention information of visual attention guidance.

[0155] The projection area ratio of the device in the user's visual field cone reflects whether the device is within the user's gaze range.

[0156] Calculation method: Establish a cone (angle of 60°~90°) with the user's head position as the vertex and the gaze direction as the axis to represent the user's visual field area. Calculate the angle between the device center point and the user's gaze direction. If it is less than the visual field angle, the device is considered to be within the visual field.

[0157] Estimate the projection weight: If the device is completely in the visual field, assign a value of 1, if it partially enters the visual field, estimate it according to the entering proportion (such as 0.7, 0.5), and if it is not in the visual field, assign a value of 0.

[0158] 3. Operation frequency weight F: Users have certain device preferences in daily use, and some devices may not be the closest in space, but if the user frequently operates them, it indicates that they have semantic bias. Using historical operation frequency as a score weighting factor helps capture user usage habits and intention tendencies.

[0159] The design of a = 0.6, b = 0.3, g = 0.1 emphasizes spatial distance as the dominant factor while taking into account user visual attention and historical preferences. This weight combination can be adaptively optimized through user behavior data to enhance the model's personalization and generalization capabilities.

[0160] S32, device state compatibility verification: for the candidate device set Compatibility judgment is made on its current state according to the control action type.

[0161] Table 2 Compatibility judgment example table

[0162]

[0163] For scenarios involving multiple device dependencies (such as "turn on humidifier" requires closing the window first), logical dependencies are loaded through predefined rules to ensure that the instruction is logically executable.

[0164] S33, intent fusion decision: when an instruction is associated with multiple compatible devices, different fusion strategies are executed according to the operation type:

[0165] 1) Environment adjustment type operation (such as temperature adjustment, light adjustment), select the highest scoring device from the candidate device set:

[0166] Indicates that in the candidate device set , select the device with the highest spatial correlation score S i as the final target device;

[0167] 2) Scene triggering type operation (such as "theater mode" "sleep mode"), execute the following fusion strategy:

[0168] Query the scene preset device group corresponding to the instruction

[0169] Iterate through all devices in the group, find the neighboring devices (within 2 meters, same type) of each device based on the spatial topology network;

[0170] Merge and remove duplicates to form the final execution device group

[0171] Indicates that the scene preset device group is merged with its neighboring device set in the topology network, and duplicate devices are removed to obtain the final execution device group ∪ represents the set operation, dedup represents the deduplication of the merged device set to prevent duplicate control, neighbors represents the neighboring device set in the topology network, which is less than 2 meters away from each device in and of the same type.

[0172] S34, instruction timing optimization: optimize the execution order of control instructions within the device group, which can be divided into the following two cases:

[0173] 1) Timing-sensitive device group: such as the combination of air conditioner and humidifier, which needs to close the window to prevent condensation; the system sorts according to the predicted response delay of the device and inserts a protection time interval:

[0174] Instruction sorting rule: sort in ascending order of response time;

[0175] Interpolation processing rule:

[0176] If there is a dependency relationship, insert a protection time interval (default 300ms);

[0177] If there is no dependency relationship, the devices are triggered in parallel.

[0178] 2) Non-timing-sensitive device group: sort the instructions according to the power / energy consumption weight, so that high-energy-consuming devices are triggered later to achieve energy-saving optimization.

[0179] S35, control signal generation: encapsulate the generated instruction sequence into device control protocol messages, adapt various protocols (such as Zigbee, Z-Wave, Matter, BLE Mesh), and send them to the target device through the intelligent gateway in sequence to complete instruction execution.

[0180] Each message contains:

[0181] Target device ID;

[0182] Action type;

[0183] Time label;

[0184] Correction or linkage flag.

[0185] As shown in Figure 2 , the smart home control system for implementing the above-mentioned smart home control method includes the following modules:

[0186] Voice acquisition and analysis module: for acquiring user voice stream and extracting semantic anchor points;

[0187] Action perception and recognition module: for acquiring user action video stream and extracting spatial anchor points;

[0188] Space-time alignment and correction module: construct a virtual time axis based on semantic anchor points, map semantic anchor points and spatial anchor points to a unified space-time coordinate system, and correct anchor points based on environmental semantic field to generate aligned instruction set;

[0189] An intent fusion and control instruction generation module is configured to fuse the alignment instruction set based on a device space topology relationship, and output a control instruction for driving the smart device.

[0190] The present application encompasses any substitutions, modifications, equivalent methods and solutions made to the essence and scope of the present application. In order to make the public have a thorough understanding of the present application, specific details are described in the following preferred embodiments of the present application, and the present application can also be fully understood without the description of these details to those skilled in the art. In addition, in order to avoid unnecessary confusion to the essence of the present application, well-known methods, processes, procedures, elements and circuits are not described in detail.

[0191] The above is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can also make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A multi-modal smart home control method, characterized in that: The following steps are involved: S1, synchronously collects the user's voice stream and action video stream, and extracts the semantic anchor points of the voice command and the spatial anchor points of the action trajectory respectively; S2, constructing a virtual timeline, mapping the semantic anchor points and spatial anchor points to a unified spatiotemporal coordinate system, and performing anchor point correction on the mapping results based on the environmental semantic field to generate an alignment instruction set; S3, performing intent fusion on the alignment instruction set according to the device space topology relationship, generating device control instructions and triggering execution.

2. The multimodal smart home control method according to claim 1, characterized in that: S1 includes collecting voice stream and action video stream through a multimodal sensor array synchronized by timestamps, and the multimodal sensor array includes: Distributed microphone groups record voice streams; The binocular depth camera captures the action video stream and generates the coordinates of the hand key points.

3. The multimodal smart home control method according to claim 1, characterized in that: The semantic anchor points of the extracted voice instructions include: Perform endpoint detection on the voice stream to segment out valid instruction segments; Identify word pairs containing action verbs and target pronouns in the instruction segment as semantic anchors; The spatial anchor point extraction of the motion trajectory includes: Track the motion trajectory of key points of the hand and calculate the rate of change of movement speed; When the speed change rate is lower than the speed change rate threshold, the coordinates of the hand key points in the current frame are used as the spatial anchor points.

4. The multimodal smart home control method according to claim 1, characterized in that: The construction of the virtual timeline includes taking the appearance moment of the semantic anchor point as the reference time point, extending forward and backward for a period of time, and constructing a semantically driven time window: In forward expansion, the time length is dynamically adjusted according to the number of words in the voice command. The more complex the command, the longer the reserved time. In the backward expansion, the time range is adjusted according to the user's historical average action response delay and its fluctuation.

5. The multimodal smart home control method according to claim 4, characterized in that: The mapping of the semantic anchor points and spatial anchor points to a unified spatiotemporal coordinate system specifically includes extracting all hand key points determined as spatial anchor points within a time window, and mapping them from the local coordinate system of the camera to a global spatial coordinate system with the home center as the origin. The mapping process combines the spatial offset relationship between the camera and the home center and the coordinate scale conversion parameters. All mapped points will be accompanied by their original timestamps to form a set of spatial anchor point sequences with time labels.

6. The multimodal smart home control method according to claim 4, characterized in that: The anchor point correction of the mapping result based on the environmental semantic field includes verifying the spatial anchor point sequence based on the real-time environmental semantic field, and performing a correction operation when it is detected that the pointing position of the spatial anchor point conflicts with the state of the environmental semantic field: (i) If the semantic anchor includes a device operation instruction and the device it points to is in an incompatible state, searching for the nearest compatible device along the device topology network; (ii) If the semantic anchor includes environmental adjustment instructions and the environmental parameters of the pointing area exceed the device calibration threshold, the spatial anchor is shifted in the direction of semantic field gradient optimization.

7. The multimodal smart home control method according to claim 6, characterized in that: The generation of the alignment instruction set includes associating the spatial anchor point and the semantic anchor point after the correction operation according to the timestamp to generate a structured instruction tuple: <action type, target position, timestamp>.

8. The multimodal smart home control method according to claim 1, characterized in that: The S3 specifically includes: S31, for each instruction tuple in the alignment instruction set, calculating a spatial correlation score between its target position and each smart device; S32, screening devices with spatial correlation scores greater than a score threshold as a candidate device set, and verifying the compatibility of the current states of the candidate devices in the candidate device set with the instruction action type; S33, Intent Fusion Decision: When a single instruction is associated with multiple compatible devices, the fusion strategy is executed: (i) If the instruction is an environmental adjustment operation, the device with the highest spatial correlation is selected for execution; (ii) If the instruction is a scenario-triggered operation, all associated devices in the topology network are activated to form a device group for collaborative execution; S34, interpolating and sorting the instruction execution timing within the device group according to the device response delay prediction value, encoding the final instruction sequence into a device executable protocol message and triggering execution.

9. The multimodal smart home control method according to claim 8, characterized in that: The spatial association score is calculated based on the Euclidean distance between the device and the target location, the projection weight of the device in the direction of the user's field of view, and the historical operation frequency weight.

10. A multimodal smart home control system, used to implement the multimodal smart home control method according to any one of claims 1 to 9, characterized in that: Includes the following modules: Voice collection and analysis module: used to collect user voice streams and extract semantic anchor points; Action perception and recognition module: used to collect user action video streams and extract spatial anchor points; Spatiotemporal alignment and correction module: This module constructs a virtual timeline based on semantic anchors, maps semantic anchors and spatial anchors to a unified spatiotemporal coordinate system, corrects the anchors based on the environment's semantic field, and generates an alignment instruction set. The intention fusion and control instruction generation module is used to perform intention fusion on the alignment instruction set based on the device space topology relationship and output control instructions for driving the smart device.

Images