Intelligent home device linkage control method and system based on multi-modal perception

Abstract

The application provides a kind of intelligent home equipment linkage control method and system based on multi-modal perception.The method includes: obtaining multi-modal perception data and device operation data, generating human event by correction, correlation and credibility evaluation;Update home state twin graph based on human event;Extract intent evidence subgraph, determine user intent and scene migration state, generate linkage control target;According to linkage control target and device capability graph, generate candidate linkage strategy, use twin graph to deduce state change trajectory;Compare state change trajectory with linkage safety envelope, determine target linkage strategy, and convert into executable linkage contract graph;Control home equipment according to executable linkage contract graph, generate linkage execution trajectory;According to the deviation and propagation range of linkage execution trajectory and state change trajectory, repair action or re-determine target linkage strategy.The application can improve multi-modal perception accuracy, enhance linkage strategy adaptability and improve execution deviation repair capability.

Description

Technical Field

[0001] The present application relates to the technical field of smart home control, and in particular to a smart home device linkage control method and system based on multi-modal perception. Background Art

[0002] With the development of the Internet of Things, artificial intelligence and edge computing technologies, smart home systems have gradually evolved from single-device remote control to collaborative control of whole-house devices. Existing smart home systems usually collect home information through voice terminals, human body sensors, camera devices and environmental sensors, and control lighting, temperature control, security and audio-visual devices according to preset instructions, conditional rules or scene templates, so as to meet users' demands for residential convenience and comfort.

[0003] However, different perception devices in the prior art mostly adopt independent data processing and identification mechanisms. There are differences in the sampling moment, spatial range and data credibility of voice, vision, human behavior and environmental state data, which are usually fused only through result splicing or fixed weights. It is difficult to accurately determine whether different perception results correspond to the same user and the same home event, and intention misjudgment is prone to occur in the case of noise interference, human body occlusion or multi-person activities. At the same time, the existing device linkage mode mainly maps scene labels directly to preset device instructions, lacks continuous characterization of user status, environmental changes, device interaction relationships and scene migration processes, and is difficult to adapt to complex home demands.

[0004] In addition, different device actions may have response delays, action coupling and execution conflicts. Existing systems usually lack pre-execution state deduction, safety boundary verification and action dependency management. When the device does not respond as expected or the environmental state deviates, the system is difficult to determine the deviation propagation range and implement local repair, and often needs to terminate the linkage scenario as a whole or re-execute it, resulting in insufficient accuracy, real-time performance and stability of device linkage. Contents of the Invention

[0005] In view of this, the embodiments of the present application provide a smart home device linkage control method and system based on multi-modal perception, so as to solve the problems existing in the prior art such as inaccurate multi-modal perception association, poor adaptability of linkage strategy, and difficult to repair execution deviation.

[0006] A first aspect of this application provides a method for the linkage control of smart home devices based on multimodal perception, comprising: acquiring multimodal perception data and device operation data within a home space; performing time-series correction, object association, and credibility assessment on the multimodal perception data to generate human-occupied events; updating a home state twin graph based on the human-occupied events, the home state twin graph representing the dynamic interaction relationships between users, home space, environmental states, and home devices; extracting an intent evidence subgraph from the home state twin graph, determining user intent and scene migration states based on the intent evidence subgraph, and generating a linkage control target; generating candidate linkage strategies based on the linkage control target and device capability graph, and deducing the state change trajectory of the candidate linkage strategies using the home state twin graph; comparing the state change trajectory with the linkage safety envelope, determining the target linkage strategy based on the comparison result, and converting the target linkage strategy into an executable linkage contract graph containing action dependencies, execution conditions, and anomaly compensation relationships; controlling home devices according to the executable linkage contract graph, generating a linkage execution trajectory; and performing local repair or re-determining the target linkage strategy for the affected device control actions based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory.

[0007] A second aspect of this application provides a smart home device linkage control system based on multimodal perception, comprising: an acquisition module for acquiring multimodal perception data and device operation data within a home space, performing time-series correction, object association, and credibility assessment on the multimodal perception data, and generating human-occupied events; an update module for updating a home state twin map based on the human-occupied events, the home state twin map representing the dynamic interaction relationships between users, home space, environmental states, and home devices; an extraction module for extracting an intent evidence sub-graph from the home state twin map, determining user intent and scene transition states based on the intent evidence sub-graph, and generating linkage control targets; and a deduction module for... The system generates candidate linkage strategies based on the linkage control target and device capability diagram, and uses a home status twin diagram to deduce the state change trajectory of the candidate linkage strategies; the comparison module compares the state change trajectory with the linkage safety envelope, determines the target linkage strategy based on the comparison result, and converts the target linkage strategy into an executable linkage contract diagram containing action dependencies, execution conditions, and anomaly compensation relationships; the generation module controls home devices according to the executable linkage contract diagram and generates linkage execution trajectories; the repair module performs local repairs on the affected device control actions or redetermines the target linkage strategy based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory.

[0008] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: By acquiring multimodal perception data and device operation data within a home space, this application performs time-series correction, object association, and credibility assessment on the multimodal perception data to generate human-occupied events. Based on these events, a home state twin map is updated, representing the dynamic interactions between users, home space, environmental states, and home appliances. An intent evidence sub-graph is extracted from the home state twin map, and user intent and scene migration states are determined based on this sub-graph, generating a linkage control target. Candidate linkage strategies are generated based on the linkage control target and device capability map, and the state change trajectory of these strategies is deduced using the home state twin map. The state change trajectory is compared with the linkage safety envelope, and the target linkage strategy is determined based on the comparison results. This target linkage strategy is then converted into an executable linkage contract diagram containing action dependencies, execution conditions, and anomaly compensation relationships. Home appliances are controlled according to the executable linkage contract diagram, generating a linkage execution trajectory. Based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory, the affected device control actions are locally repaired or the target linkage strategy is redefined. This application can improve the accuracy of multimodal perception, enhance the adaptability of linkage strategies, and improve the ability to correct execution deviations. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the smart home device linkage control method based on multimodal perception provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the smart home device linkage control system based on multimodal perception provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0012] Existing smart home systems typically acquire home information through voice terminals, visual acquisition devices, human body sensing devices, and environmental sensors. Based on preset commands, conditional rules, or scene templates, they control home appliances such as lighting, temperature control, security, and audio-visual systems. Although some systems introduce multimodal perception and intelligent decision-making models, the data from different modalities are usually independent of each other. Device interaction is still mainly based on the mapping between scene tags and fixed control commands, lacking a unified model of the dynamic relationships between users, space, environment, and devices.

[0013] In existing technologies, multimodal data exhibits differences in sampling time, sensing range, recognition accuracy, and data reliability. Simple splicing or fixed-weight fusion is insufficient to accurately determine whether different sensing results originate from the same user and the same home event, easily leading to misinterpretations of intent under noise interference, human occlusion, or multi-person activity. Furthermore, fixed scene templates struggle to adapt to continuous changes in user states and home scenarios, and fail to adequately consider the coupling, response latency, and execution conflicts between different device actions. When device execution malfunctions or environmental responses deviate from expectations, existing systems struggle to track the propagation range of deviations and implement targeted repairs. Therefore, this application aims to address the problems of inaccurate multimodal sensing associations, poor adaptability of linkage strategies, and difficulty in repairing execution deviations.

[0014] To address the aforementioned issues, this application first acquires multimodal perception data and device operation data within the home space. It then generates residential events through time-series correction, object association, and credibility assessment, and updates a residential state twin map representing the dynamic interactions between users, space, environment, and home appliances based on these events. Subsequently, it extracts an intent evidence sub-graph from the residential state twin map to determine user intent and scene transition states, thereby generating a linkage control objective. Based on the linkage control objective and device capability map, candidate linkage strategies are generated, and the residential state twin map is used to deduce the state change trajectory corresponding to each candidate linkage strategy.

[0015] Furthermore, this application compares the state change trajectory with the linkage safety envelope, determines the target linkage strategy from the candidate linkage strategies, and converts the target linkage strategy into an executable linkage contract diagram that includes action dependencies, execution conditions, and anomaly compensation relationships. Home appliances are controlled according to the executable linkage contract diagram, and a linkage execution trajectory is generated. Then, based on the deviation between the linkage execution trajectory and the state change trajectory and its propagation range, the control actions of the affected devices are locally repaired. When the local repair cannot meet the control requirements, the target linkage strategy is re-determined based on the current home state.

[0016] Through the above technical solutions, this application can improve the correlation accuracy of asynchronous and heterogeneous multimodal sensing data, enhance the reliability of user intent and scene transition status recognition; generate and deduce linkage strategies based on the current home status and dynamic interaction relationships of devices, improve the adaptability and security of cross-category device linkage; and implement local repair or strategy reconstruction according to the propagation range of execution deviation, thereby improving the real-time performance, stability and fault tolerance of smart home device linkage control.

[0017] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0018] Figure 1 This is a flowchart illustrating the smart home device linkage control method based on multimodal perception provided in an embodiment of this application. Figure 1 As shown, the method may specifically include: S101: Acquire multimodal sensing data and equipment operation data within the home space, perform time-series correction, object association, and credibility assessment on the multimodal sensing data, and generate human settlement events; S102, Update the home status twin based on human settlement events. The home status twin represents the dynamic interaction between users, home space, environmental status and home equipment. S103, extract the intent evidence sub-graph from the home status twin graph, determine the user intent and scene transition status based on the intent evidence sub-graph, and generate linkage control target; S104, Generate candidate linkage strategies based on linkage control targets and equipment capability diagrams, and use the home status twin diagram to deduce the state change trajectory of the candidate linkage strategies; S105, compare the state change trajectory with the linkage safety envelope, determine the target linkage strategy based on the comparison result, and convert the target linkage strategy into an executable linkage contract diagram that includes action dependencies, execution conditions and exception compensation relationships; S106, Control home appliances according to the executable linkage contract diagram and generate linkage execution trajectory; S107, based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory, perform local repairs on the affected equipment control actions or redetermine the target linkage strategy.

[0019] In some embodiments, multimodal sensing data and device operation data within a home space are acquired; the multimodal sensing data undergoes time-series correction, object association, and credibility assessment to generate human settlement events, including: Based on a unified time reference and a home space semantic reference, multimodal perception data is time-aligned and spatially mapped to generate modal events that represent state changes. The perception credibility is determined based on the data quality, perception continuity, and spatial matching degree of each modal event; Based on user objects, spatial objects, and associated time windows, cross-modal associations are performed on modal events and device operation data to generate candidate habitation events; The modal conflict degree is determined based on the spatiotemporal consistency, perceived credibility, and state continuity of each modal event in the candidate habitation event. Based on the modal conflict degree, candidate habitation events are fused or candidate states are retained to generate habitation events.

[0020] Specifically, the home edge control node pre-establishes a unified time reference and a semantic reference for the home space. The unified time reference can be periodically sent by the edge control node to the voice acquisition unit, vision acquisition unit, human posture acquisition unit, environmental sensors, and device gateway, and the transmission delay and processing delay of each acquisition link are recorded. For low-power sensors that cannot be directly synchronized, the clock offset can be estimated based on the time of adjacent state changes and the time of gateway reception, and the timestamps of each data can be converted to the unified clock.

[0021] The semantic baseline for home space is established based on room boundaries, functional areas, equipment installation locations, and sensor coverage. This divides the home space into identifiable spatial units, and assigns spatial identifiers, adjacency relationships, and sensor coverage relationships to each unit. Edge control nodes perform resampling or event-triggered segmentation on continuous data according to a unified time baseline. They then map the sound source direction, human coordinates, posture direction, environmental measurement points, and equipment locations to corresponding spatial units according to the spatial semantic baseline. This generates modal events such as speech semantic changes, human entry, posture changes, environmental abrupt changes, and equipment status changes. Each modal event is configured with an event identifier, modal source, occurrence time, spatial identifier, state change amount, and original quality parameters.

[0022] Perceived credibility is used to characterize the reliability of a single modal event in truly reflecting changes in home status. For the speech modality, a data quality score can be generated based on the effective speech ratio, signal-to-noise ratio, sound source direction stability, and semantic recognition probability. For the visual or posture modality, a perceived continuity score can be generated based on the completeness of the human target, the number of consecutive tracking frames, the proportion of missing key points, and the amount of positional drift. For the environmental modality, a data quality score can be generated based on the sensor self-check status, the reasonableness of the rate of change, and the consistency between adjacent measurement points.

[0023] Spatial matching degree can be determined by the overlap between the event location area and the effective coverage area of ​​the sensor, the angle between the sound source direction and the human body position, and the distance between the device's operating area and the area of ​​environmental change. The edge control node unifies and normalizes the above scores and aggregates them according to the basic weights configured for each modality type, obtaining a perception confidence level within the range of 0 to 1. If a modality experiences a short-term loss, the confidence level can be attenuated and compensated using the continuity of preceding and following events, rather than directly deleting the corresponding event.

[0024] In cross-modal association, edge control nodes first determine candidate user objects based on human tracking identifiers, voiceprint features, or terminal binding information, then determine candidate spatial objects based on spatial identifiers, and establish an association time window centered on the event occurrence time. The association time window can be adaptively adjusted according to the modal response delay; for example, the window for voice events and gesture events is set to 2 seconds, and the window for human entry events and illumination change events is set to 5 seconds. For modal events that fall within the same association time window and have the same user object and spatial object, time proximity, spatial proximity, and state causality fit are calculated, and concurrent device operation data is associated to generate candidate residential events. State causality fit is used to determine whether a change in device state may be caused by user behavior or environmental changes; for example, a high fit occurs when a gesture is directed towards a curtain and the curtain subsequently begins to close; a low fit occurs between an air conditioner temperature change and human actions away from the air conditioner's operating area.

[0025] For example, taking a living room scene at night, the edge control node detects user A entering the sofa area at 21:03:12, recognizes user A performing a closing gesture towards the window at 21:03:14, and collects voice semantics indicating reduced brightness at 21:03:15. Simultaneously, it detects that the living room illuminance is 420 lx, the curtains are open, and the audio-visual equipment is in standby mode. After time correction, all events fall within the same associated time window; the human body position, gesture direction, and sound source direction are all mapped to the living room sofa area, ensuring consistent user identification. The system obtains a confidence level of 0.94 for the human body entry event, 0.88 for the gesture event, and 0.81 for the voice event, and generates candidate residential events for the "movie viewing preparation" category based on the curtain status.

[0026] Modal conflict degree is determined jointly based on temporal deviation, spatial deviation, semantic consistency, perceptual credibility difference, and state continuity. When speech semantics, gesture direction, and current device state support each other, the modal conflict degree is low. The system merges candidate events according to the fusion weight after credibility normalization and outputs a human-occupied event that includes user A, living room sofa area, intention to reduce ambient brightness, associated device state, and comprehensive credibility.

[0027] If the sound of the TV is misinterpreted as the curtains being opened, and the user's gesture continues to point in the closing direction, and the curtains' historical operation state is consistent with the gesture, then the voice event has a high degree of conflict with other events. The system reduces the contribution of the voice and retains two candidate states: the curtains are closed and the curtains are open, for further verification through subsequent intent evidence inference. Thus, without relying on the results of a single sensor, it can form residential events with unified spatiotemporal identification, object attribution, and credibility description, improving the accuracy and anti-interference capability of multimodal perception association in complex home environments.

[0028] In some embodiments, updating the home status twin based on residential events includes: Based on the user objects, space objects, and device objects associated with human settlement events, the event state is mapped to the corresponding state node in the home state twin graph; Based on the status changes, perceived credibility, and occurrence sequence of human settlement events, update the current status, change trend, and status credibility of the corresponding status nodes; Based on the interaction relationships within the home space, determine the state association edges associated with the update nodes, and adjust the direction, intensity, and response latency of the state association edges according to the state differences before and after the event. Propagate state changes along the corrected state-related edges, update affected state nodes, and form a home status twin graph representing the current home status and its changing trends based on the propagation results.

[0029] Specifically, a home status twin graph can be deployed at the home edge control node to continuously represent the dynamic relationships between human activities, space occupancy, environmental changes, and equipment operation in a graph structure. The graph includes user nodes, space nodes, environment nodes, and device nodes. Each node is configured with an object identifier, current state, state timestamp, change trend, and state reliability. Occupancy association edges, spatial adjacency edges, device action edges, environmental propagation edges, and device coupling edges are set between nodes. Each state association edge is configured with action direction, action strength, response delay, effective range, and edge reliability. During initial graph construction, a basic graph structure can be established based on the residential spatial layout, equipment installation locations, sensor coverage areas, and equipment functional parameters. Historical operating data can be used to estimate the initial action parameters of equipment actions on relevant states.

[0030] Furthermore, upon receiving a human activity event, the edge control node parses the user identifier, space identifier, device identifier, state change, and occurrence time carried by the event, and maps them to the corresponding state nodes. For user activity events, the human position, posture state, and activity intensity are written to the corresponding user node, and the occupancy association edge between the user node and the current space node is updated; for environmental change events, the detected value and change amount are written to the corresponding environment node; for device operation events, the operating mode, control parameters, and execution status are written to the corresponding device node. If an event contains multiple associated objects, multiple nodes are updated synchronously according to the object ownership relationship in the event, and the association between this update is saved through the event identifier to avoid the same event being written repeatedly.

[0031] When updating node states, not only are the latest event values ​​used to overwrite the original states, but a recursive fusion is also performed, combining the direction of state change, perceived reliability, and event timing. Specifically, a time decay weight is set based on the interval between the event occurrence time and the node's original state time, an observation weight is set based on perceived reliability, and the current state is obtained by combining the rate of change in the previous period. For continuous, stable events with high reliability, the update ratio of the current observation value is increased; for events with low reliability or significantly inconsistent with historical trends, the update ratio is decreased and the original state is retained. The trend of change can be determined based on the slope, fluctuation amplitude, and duration of state values ​​within a continuous time window, while the state reliability is dynamically corrected based on the reliability of the corresponding event, the consistency of continuous observations, and data missingness.

[0032] The updating of state-related edges is based on the differences in node states before and after an event and the actual interaction relationships within the home space. Edge control nodes first search for state-related edges connected to the updated node in the base graph structure, then compare the state changes of the source node and the response changes of the target node before and after the event. When the changes in the source node and the target node are in the same direction and the temporal relationship is stable, the strength and reliability of the corresponding related edges are increased; when the changes are in opposite directions, the direction of interaction is corrected to negative; when the target node does not respond within the expected time, the response delay is extended or the strength of interaction is reduced. To avoid parameter drift caused by occasional disturbances, a sliding time window can be used to aggregate multiple events, and state-related edges are corrected only when the accumulated evidence meets the update conditions.

[0033] When propagating state changes along the edges associated with the corrected state, the currently updated node is used as the starting node. The expected impact on adjacent nodes is calculated based on edge credibility, influence strength, and response latency, and propagated layer by layer according to spatial adjacency and device range. A maximum propagation layer and influence attenuation conditions are set during propagation. Propagation stops when the cumulative influence falls below the propagation threshold or exceeds the device's effective range. For target nodes simultaneously affected by multiple source nodes, weighted aggregation can be performed based on the influence direction and credibility of each propagation path, and conflicting candidate states are retained for subsequent intent identification and strategy deduction.

[0034] For example, when user A enters the living room sofa area at night, the system receives occupancy events such as human presence, stable sitting posture, curtain closure, and reduced main light brightness. The edge control node maps user A's location and activity status to the user node, the living room occupancy status to the space node, the curtain closure status and main light dimming status to the corresponding device nodes, and the change in illuminance from 420 lx to 180 lx to the living room illuminance node. Because human tracking is continuous, device feedback is complete, and the illuminance change is as expected, the reliability of each node's status is updated to a high level.

[0035] The system further compares the sequential relationship between curtain closure, main light activation, and illuminance changes, determining that both curtain closure and main light dimming negatively impact living room illuminance. It then adjusts the intensity and response delay of the two device action edges based on the actual response amplitude. Subsequently, the illuminance change propagates along the spatial adjacent edges to the living room passageway. The system predicts that the illuminance in the passageway may drop to the lower limit of the safe range and updates the change trend of the corresponding environmental nodes. The resulting home status twin simultaneously represents user A in a living room leisure state, a continuous decrease in living room illuminance, curtains and main light in a linked state, and the passageway potentially being affected. This provides a continuous and reliable state foundation for subsequent extraction of intent evidence, generation of linked control targets, and deduction of device actions.

[0036] In some embodiments, an intent evidence subgraph is extracted from the home status twin graph, and the user intent and scene transition status are determined based on the intent evidence subgraph to generate a linkage control target, including: Based on the current state nodes associated with human settlement events, extract the state change path along the dynamic interaction relationship and construct an intention evidence subgraph; Based on the intent evidence from different sources and their corresponding credibility in the intent evidence subgraph, the evidence support for the candidate user's intent is determined; By combining the temporal evolution of candidate user intents with their matching relationship with historical scene migration patterns, user intents, scene migration states, and intent uncertainty are determined. Based on user intent, scene migration status, and intent uncertainty, the current home status is mapped to the target scene status to generate a linkage control target that limits the target state range, response constraints, and device scope.

[0037] Specifically, the home edge control node pre-establishes a user intent database, a scene state database, and a historical scene migration pattern database. The user intent database records candidate intents such as resting, watching movies, meeting guests, leaving home, and preparing for sleep, along with their corresponding behavioral, environmental, and device evidence. The scene state database records the permissible ranges of personnel location, activity intensity, illumination, temperature and humidity, noise, and device operating status in each scene. The historical scene migration pattern database stores the migration order, duration, and triggering conditions between common scenes in different time periods according to user identifiers. After the current residential event is written into the home state twin graph, a restricted path search is performed along the occupancy association edge, spatial adjacency edge, device action edge, and environmental propagation edge, using the user node, space node, environmental node, and device node directly associated with the event as seed nodes.

[0038] During path extraction, a backtracking time window and a forward observation window can be set based on the event occurrence time. Within the backtracking time window, the preceding changes that led to the current state are extracted, and within the forward observation window, potential state responses that may continue to evolve are extracted. Edge control nodes calculate path relevance based on the direction, intensity, response delay, and edge credibility of state-related edges, and filter out paths that exceed spatial range, time span, or relevance thresholds. When multiple paths converge to the same state node, paths that can explain the current human-occupied event and do not have obvious temporal contradictions are retained. The state node, state-related edges, event identifier, and credibility in the path are combined into an intent evidence subgraph. The intent evidence subgraph retains both the current observation results and the evolutionary relationship between the observation results and historical states, thereby avoiding direct determination of user intent based solely on a single voice or single action.

[0039] The evidentiary support for candidate user intents can be determined through evidence matching and credible aggregation. Edge control nodes match state changes in the intent evidence subgraph with intent evidence templates in the user intent database to obtain the support direction and matching degree of each piece of evidence for different candidate intents. Human position, posture, and movement direction can form behavioral evidence; changes in illuminance, temperature, and noise can form environmental evidence; and the status of curtains, lamps, air conditioners, and audio-visual equipment can form equipment evidence. Redundancy removal is performed on highly correlated evidence from the same source to avoid duplicate accumulation; the joint contribution is increased for evidence from different sources that corroborate each other. Subsequently, the evidence matching degree is multiplied by the node state credibility, associated edge credibility, and event freshness to obtain the contribution of individual evidence, and the evidentiary support for candidate intents is calculated based on the difference between supporting evidence and conflicting evidence.

[0040] When determining the scene migration status, edge control nodes form a candidate intent support sequence according to the event timeline, determine whether the support is continuously increasing, stably maintained, or rapidly decaying, and perform sequence matching with historical scene migration patterns. The matching content includes the starting scene, target scene, order of key state changes, migration duration, and device response sequence. If the current sequence meets the requirements for path similarity and time adaptation with a certain historical pattern, the corresponding user intent and scene migration status are determined. If multiple candidate intents have similar support, or key evidence is missing or conflicting evidence persists, multiple candidate intents are retained, and intent uncertainty is generated based on support interval, evidence completeness, and differences in historical pattern matching.

[0041] For example, at 9:03 PM, User A enters the living room sofa area and remains seated. The curtains change from open to closed, the main light intensity decreases, and the audio-visual equipment remains in standby mode. Using User A, the living room space, the curtains, the main light, and the intensity node as seeds, the system extracts the state change path from the user entering the living room, stabilizing their posture, closing the curtains, to the environment darkening, forming an intent evidence subgraph. Both the behavioral and device evidence support the intent to prepare for watching a movie, while the user not leaving the living room or turning off all the lights does not support the intent to leave home or sleep. Through trusted aggregation, the support for the intent to prepare for watching a movie is 0.86, the support for the intent to meet guests is 0.38, and the support for the intent to rest is 0.44. The current sequence of changes matches User A's historical migration pattern of first sitting down, then closing the curtains, and finally starting the audio-visual equipment. The system determines that the scene is transitioning from a daily activity state to a movie-watching state and sets the intent uncertainty at a low level.

[0042] When generating linkage control targets, the system reads the target state corresponding to the viewing scene from the scene state library, compares the current home state with the target scene state node by node, and obtains state differences such as excessive illuminance, audio-visual equipment not started, and air conditioning status meeting requirements. For state nodes with differences, a target state range is set; for example, the illuminance in the sofa area is limited to 80lx to 150lx, and the audio-visual equipment is limited to a playable state. Response constraints are set based on intent uncertainty: when the intent is clear, low-risk actions are allowed to be executed directly; when the intent is uncertain, high-impact actions are restricted and confirmation conditions are retained. The system then determines the device's effective range based on the device's active edge and spatial coverage relationship to prevent unrelated devices such as bedroom lights and kitchen appliances from participating in the linkage.

[0043] This generates a linkage control target that simultaneously defines the target state range, response timing, safety conditions, and device scope, providing clear constraints for subsequent candidate linkage strategy generation and state deduction, and improving the adaptability of user intent determination and scenario linkage.

[0044] In some embodiments, candidate linkage strategies are generated based on the linkage control target and the device capability map, and the state change trajectory of the candidate linkage strategies is deduced using a home status twin map, including: The linkage control objective is decomposed into objective state changes, and candidate equipment actions that can drive the objective state changes are matched along the equipment action relationship in the equipment capability diagram. The action adaptability is determined based on the scope of action, response latency and action constraints of the candidate device actions, and the candidate device actions are combined based on the dependencies and coupling relationships between devices to generate candidate linkage strategies; Map the device actions corresponding to each candidate linkage strategy to the home status twin graph, propagate the impact of actions according to the dynamic interaction relationship, and predict the temporal changes of relevant status nodes. By aggregating device status changes and environmental responses, the system generates status change trajectories corresponding to each candidate linkage strategy.

[0045] Specifically, the device capability diagram is used to characterize the control actions that each home appliance can generate, the objects of action, and the collaborative constraints between devices. It can be pre-built by the home edge control node based on device registration information, spatial installation information, historical response data, and operating rules. Device nodes in the capability diagram record device type, location, current status, available capabilities, control range, and communication status. Action nodes record executable actions such as switching, mode switching, and parameter adjustment. Action nodes are connected to changeable environmental state nodes through device interaction relationships. Each device interaction relationship is configured with action direction, action range, response delay, state change amplitude, energy consumption cost, and capability reliability. Prerequisite dependencies, synchronization relationships, mutual exclusion relationships, and action coupling relationships can also be set between device nodes.

[0046] Furthermore, after obtaining the linkage control target, the edge control node decomposes the linkage control target into multiple target state changes according to the target state type, target state range, completion time limit, and target space. For example, the movie-watching linkage control target is decomposed into target state changes such as reducing the illuminance of the sofa area, reducing the entry of external light, starting audio-visual playback, and maintaining safe lighting in the passage area. For each target state change, the corresponding environmental state node is used as the starting point for retrieval, and action nodes with effective interaction relationships are searched backward along the device capability map. Device actions that can directly drive the target state change are filtered, and device actions that can indirectly produce effects through device coupling or spatial propagation are further searched. If a device is currently offline, in a faulty state, or has insufficient remaining capacity, the corresponding action node is deleted; if the target state change can be achieved by multiple devices, the actions of different devices are retained as a set of candidate actions.

[0047] Action adaptability is determined based on the coverage of the action range, predicted response latency, state adjustment accuracy, current device availability, and the degree to which action constraints are met. For actions whose action range highly overlaps with the target space, have low response latency, and whose adjustment range covers the target state range, action adaptability is increased; for actions that may affect unrelated spaces, whose action amplitude is difficult to finely adjust, or that require frequent start-stop operations, action adaptability is decreased. Action constraints may include upper and lower limits of device parameters, minimum start-stop intervals, continuous runtime, silent period restrictions, and user-prohibited control conditions. The edge control node normalizes each adaptability factor, sets weights according to the importance of the target state, and obtains the action adaptability corresponding to each candidate device action.

[0048] When generating candidate linkage strategies, the system combines candidate device actions based on the dependencies and coupling relationships between devices. When there are prerequisite dependencies, preparatory actions are placed before the target action; for example, power supply status is checked before starting audio-visual equipment. When there are mutually exclusive relationships, conflicting actions are not placed in the same strategy. When there are synergistic effects, the adjustment range is allocated according to the contribution of each action to avoid multiple devices simultaneously over-adjusting the same environmental state. The system can use a constraint search method to select at least one action from the set of candidate actions for each target state change and verify whether the action combination can cover all target state changes, forming multiple sets of candidate linkage strategies.

[0049] After each candidate linkage strategy is generated, the edge control node maps device actions to device nodes in the home status twin graph and sets the expected state changes of the device nodes according to the action plan. Then, the action impact is propagated along the device action edge, environment propagation edge, and spatial adjacency edge time-by-time. During propagation, the state change amplitude is calculated based on the action intensity, the start time of the state change is determined based on the response delay, the impact is attenuated based on spatial distance and propagation level, and the prediction results are corrected by combining the current environmental state, user activity state, and other device operating states. When multiple actions act on the same state node, actions in the same direction are superimposed and limited to a reasonable state range, actions in opposite directions are calculated to cancel each other out, and actions with strong coupling are predicted by calling joint response parameters.

[0050] For example, when user A transitions from daily activities to movie watching, the linkage control objective requires reducing the illuminance in the sofa area from 420 lx to 80 lx-150 lx within 30 seconds, closing the external light channel, starting the audio-visual equipment, and simultaneously maintaining the illuminance in the living room passageway at no less than 60 lx. In the equipment capability diagram, the main light dimming action, the floor lamp dimming action, and the curtain closing action can all reduce the illuminance in the sofa area; the corridor light turning on action can compensate for the illuminance in the passageway; and the audio-visual equipment starting action can complete the playback state switch. The system calculations show that the curtain closing action has a large range of action and a response delay of 8 seconds, the main light dimming action has a fast response and high adjustment accuracy, and the floor lamp turning off action may make the passageway too dark, thus reducing its action adaptability.

[0051] Based on action adaptability and device relationships, the system generates the first candidate linkage strategy, which includes closing the curtains, adjusting the main light to 25% brightness, turning on the hallway light, and starting the audio-visual equipment; the second candidate linkage strategy includes closing the curtains, turning off the main light, keeping the floor lamp at a low brightness, and starting the audio-visual equipment; the third candidate linkage strategy includes only lowering the main light and starting the audio-visual equipment. After mapping the three strategies to a home status twin map, the system predicts that the first strategy can stabilize the illuminance in the sofa area at 120 lx after 12 seconds, and maintain the illuminance in the passageway area at 68 lx; the second strategy may reduce the illuminance in the passageway area to 48 lx; the third strategy is affected by external light, and the illuminance in the sofa area can only decrease to 190 lx.

[0052] The system collects data on curtain position, lighting brightness, audio-visual equipment status, and illuminance changes in each space according to the execution sequence, forming state change trajectories that include equipment state sequences, environmental response sequences, and the process of achieving the target. Through this processing, the abstract linkage control target can be transformed into candidate linkage strategies with clear combinations of equipment actions and predicted response processes, providing a verifiable trajectory basis for subsequent safety envelope comparison and target strategy selection.

[0053] In some embodiments, the state change trajectory is compared with the linkage safety envelope, the target linkage strategy is determined based on the comparison result, and the target linkage strategy is converted into an executable linkage contract graph containing action dependencies, execution conditions, and exception compensation relationships, including: Based on the state nodes and evolution sequence, the trajectory of each state change is matched with the state boundary, change rate constraint and device interlock constraint in the linkage safety envelope to determine the risk of trajectory going out of bounds; Candidate linkage strategies that do not meet the safety conditions and have the risk of trajectory going out of bounds are screened out. The target linkage strategy is determined based on the degree of goal achievement, response cost and action stability of the remaining candidate linkage strategies. Generate action nodes and action dependencies according to the action sequence and state dependency of each device in the target linkage strategy; Configure execution conditions for each action node based on device and environment status, and establish anomaly compensation relationships based on the corresponding alternative actions and fallback paths for action failures or state deviations, generating an executable linkage contract diagram.

[0054] Specifically, the linkage safety envelope is used to limit the permissible device and environmental states during the execution of candidate linkage strategies. It can be dynamically generated by the home edge control node based on the current home status, device capability boundaries, spatial safety rules, and historical anomaly samples. The linkage safety envelope is organized according to state nodes and time slices. Each time slice records the permissible range of environmental states such as temperature, illuminance, noise, air quality, and space occupancy, as well as the permissible range of device on / off status, operating parameters, and duration. It also configures the maximum change per unit time, cumulative change, and device interlocking conditions. For spaces with children, the elderly, or nighttime access needs, the lower limit of illuminance, the rate of temperature change, and the magnitude of device actions can be tightened. For nodes with low state reliability, the prediction uncertainty range can be expanded, and the risk assessment margin can be increased.

[0055] When comparing state change trajectories, edge control nodes first align the state sequence in the trajectory with the linked safety envelope item by item, according to device identifier, environmental state identifier, and prediction time. State boundary risk can be determined based on the magnitude and duration of the predicted value exceeding the allowable range; change rate risk can be determined based on the ratio of the state difference between adjacent time slices to the maximum allowable change; and device interlock risk is determined based on the action overlap time, preceding state, and combination of mutually exclusive devices. For state nodes with uncertain ranges in the prediction results, the degree of overlap between the uncertain range and the safety boundary can also be calculated. When the predicted center value does not exceed the boundary but the uncertain range significantly exceeds the boundary, the trajectory boundary violation risk is still increased. The system normalizes various risks and combines the importance of the state node, its spatial occupancy status, and the duration of the risk to form a comprehensive risk value.

[0056] The selection of candidate linkage strategies employs a combination of hard and soft constraints. If the trajectory triggering device is forcibly interlocked, the critical spatial state crosses the prohibited boundary, or multiple consecutive time slices fail to meet safety conditions, the corresponding strategy is directly eliminated. If it only approaches the warning boundary, the strategy is retained, and the risk margin is recorded. For the remaining strategies, the degree of goal achievement can be determined based on the proportion of each target state entering the target range, the completion time, and the duration of maintenance. The response cost can be determined by comprehensively considering the number of device start-ups and shutdowns, energy consumption increments, communication frequency, and user interference levels. The stability of the action can be determined based on frequent action reversals, device state oscillations, and environmental response fluctuations. Edge control nodes are prioritized according to the safety principle. Under the premise that the overall risk requirements are met, strategies with high goal achievement, low response cost, and good action stability are selected as the target linkage strategies.

[0057] Furthermore, once the target linkage strategy is determined, the system converts each device action into an action node and establishes directed action dependency edges based on state dependencies. Action nodes can record the target device, control parameters, planned execution time, maximum waiting time, and completion criteria. If a subsequent action must be executed after a preceding action is completed, a completion dependency edge is established; if multiple actions need to be started synchronously, a synchronization dependency edge is established; if an action is only executed after the environment state reaches an intermediate range, a state trigger dependency edge is established. Through these dependencies, the originally parallel control instructions can be transformed into a graph structure with a clear execution order and conditional constraints.

[0058] Each action node also needs to be configured with execution conditions. These conditions may include whether the device is online, the device's current mode allows switching, the target space is occupied or unoccupied, the environmental state is within the allowed starting range, and the preceding actions have been completed. Before execution, the edge control node reads device feedback and environmental perception results to verify the conditions; the action node is only activated when all necessary conditions are met. For potential device rejection, communication timeouts, incomplete actions, and environmental response deviations, the system configures alternative actions and fallback paths for the action nodes. Alternative actions are selected from other device actions that can achieve the same target state change and have a similar scope of action; fallback paths are used to undo executed actions or restore to a safe intermediate state. The exception compensation relationship also records triggering conditions, compensation priorities, and the maximum number of retries to prevent exception handling from becoming a loop.

[0059] For example, in a living room movie-watching scenario, the first candidate strategy predicts that the illuminance in the sofa area will drop to 120 lx while the illuminance in the passageway area will remain at 68 lx, and the actions of each device will meet the interlocking requirements. The second candidate strategy predicts that the illuminance in the passageway area will drop to 48 lx, which is below the 60 lx safety limit. The third candidate strategy predicts that the illuminance in the sofa area will remain at 190 lx, which does not achieve the movie-watching goal. The system eliminates the second candidate strategy and compares the goal achievement degree and response cost of the first and third candidate strategies, ultimately selecting the first candidate strategy. Subsequently, action nodes such as closing the curtains, adjusting the main light to 25%, turning on the corridor light, and starting the audio-visual equipment are generated. The system is set to allow the curtains to close and the main light to dim in parallel, the corridor light to turn on must be triggered when the illuminance in the passageway area is below 75 lx, and the audio-visual equipment to start must be executed after someone is in the sofa area and the illuminance is within the target range.

[0060] If the curtains do not return to the closed state within 10 seconds, the system maintains the main light brightness at no less than 35% according to the anomaly compensation relationship, and further adjusts the main light to complete the viewing illumination adjustment; if the corridor light is offline, it reverts to the main light zone lighting mode to restore the illumination of the passage area to a safe range. The final executable linkage contract diagram can uniformly link trajectory safety verification, action sequence, state conditions, and anomaly handling, so that the target linkage strategy has verifiable security before execution and clear alternative and fallback paths in case of execution anomalies, thereby improving the stability and fault tolerance of device linkage.

[0061] In some embodiments, the linkage security envelope is generated in the following manner: Extract the state nodes and dynamic interaction relationships associated with the linkage control target from the home status twin diagram; Based on the current state, change trend, and state reliability of the state nodes, and combined with the action constraints in the device capability diagram, determine the allowable state range and allowable rate of change. Propagate the expected impact of the equipment actions corresponding to the candidate linkage strategies along the dynamic interaction relationship, and determine the equipment interlock boundary and cross-space impact boundary; The permissible state range, permissible rate of change, device interlock boundaries, and cross-space impact boundaries are associated according to the evolution sequence to generate a linked safety envelope that is dynamically updated with the current home status.

[0062] Specifically, the linkage safety envelope is not a pre-fixed set of thresholds, but rather dynamically formed by the home edge control node in conjunction with the current home status, device capabilities, and the impact propagation process of candidate actions. After receiving the linkage control target, the edge control node first determines the target space, target environmental parameters, and candidate device objects from the target status range and device range. Then, starting from the corresponding status node, it extracts the nodes and dynamic relationships related to the achievement of the target and the safety impact from the home status twin graph.

[0063] The extraction scope can include user nodes, spatial nodes, environment nodes, and device nodes within the target space, as well as peripheral nodes connected to the target node through spatial adjacency edges, device action edges, and environment propagation edges. For associations with action strength below the impact threshold, response delay exceeding the linkage cycle, or insufficient path reliability, further expansion can be stopped, thereby forming a safety constraint subgraph for this linkage task.

[0064] The permissible state range is determined jointly based on the node's current state, its changing trend, state reliability, and scene safety rules. For continuous states such as illuminance, temperature, humidity, and noise, a basic safety range can be read first, and then adjusted or shifted according to space usage, personnel occupancy, and the target scene. For example, the lower limit of illuminance can be increased when people are passing through, the upper limit of noise can be decreased when people are resting at night, and the temperature adjustment range can be reduced when children are in the target space.

[0065] In some examples, if the node status has high reliability and a stable trend, a smaller safety margin can be used within the basic safety range; if data is missing, observations conflict, or the status fluctuates rapidly, the prediction error range should be expanded, and the executable range should be tightened accordingly. For equipment status, the allowable ranges for switch status, operating mode, and control parameters can be formed by combining the rated parameters, minimum start-stop interval, continuous operating time, and available adjustment levels in the equipment capability diagram.

[0066] The permissible rate of change limits the magnitude of state changes between adjacent time slices. Edge control nodes can determine upper limits for temperature change rate, illuminance change rate, noise increment, and equipment power change rate based on the device's historical response curves, user comfort constraints, and spatial inertia parameters. For nodes whose states are close to the boundary or whose change direction continuously points towards the boundary, the permissible rate of change is further reduced; for nodes whose states are far from the boundary and whose response is slow, the rate limit can be appropriately relaxed. Equipment action constraints simultaneously limit the frequency of action initiation, parameter jumps, and the number of continuous adjustments to avoid frequent starts and stops or large jumps that could cause equipment oscillations and user discomfort.

[0067] When determining device interlock boundaries, the system maps actions in candidate linkage strategies to a safety constraint subgraph and propagates the expected impact along dynamic interaction relationships. If two actions impose incompatible control parameters on the same device, or if the simultaneous operation of multiple devices would cause power supply capacity overruns, communication resource contention, or functional conflicts, a prohibited concurrency boundary is established. If subsequent actions must wait for preceding devices to reach a stable state, a time-based interlock boundary is established. If the device is in protection mode, manually locked state, or maintenance state, an unexecutable boundary is established. Interlock boundaries can be described by action combinations, triggering conditions, durations, and release conditions, enabling the system to identify parallel, sequential, and prohibited combinations of device actions.

[0068] Cross-spatial influence boundaries are used to constrain the spillover effects of device actions on non-target spaces. The system predicts the magnitude and arrival delay of candidate actions on the state of adjacent spaces based on spatial adjacency, airflow paths, sound and light propagation paths, and device coverage. It also sets allowable spillover ranges based on the occupants' status and safety requirements in the adjacent spaces. For example, adjusting living room lighting may affect corridor illuminance, air conditioning may change the temperature in the adjacent dining room, and audio-visual equipment volume may increase bedroom noise. If the predicted impact exceeds the allowable range for adjacent spaces, the corresponding action magnitude, duration, or execution order will be limited within the cross-spatial influence boundary.

[0069] Edge control nodes discretize the allowed state range, allowed rate of change, device interlock boundaries, and cross-spatial impact boundaries into continuous time slices according to the expected execution sequence of candidate linkage strategies. For each time slice, a state node, constraint type, boundary value, confidence level, and risk margin are associated, forming a linkage safety envelope. When the home status is updated, the user's location changes, or the device's availability changes, the system recalculates the affected time slices without needing to rebuild all boundaries, allowing the linkage safety envelope to update dynamically with the current state.

[0070] For example, when user A enters the movie-watching preparation state in the living room, the system extracts the illuminance of the sofa area, the illuminance of the passageway area, the position of the curtains, the brightness of the main light, the status of the corridor lights, the status of the audio-visual equipment, and the light propagation relationship between the living room and the corridor. Based on the user's location in the sofa area and the continued need for nighttime passage, the allowable range for illuminance in the sofa area is set to 80 lx to 150 lx, the lower limit for illuminance in the passageway area is set to 60 lx, and the rate of illuminance decrease per unit time is limited to avoid sudden visual changes.

[0071] The system further determined that curtain closing and main light dimming can be performed in parallel, audio-visual equipment startup must be performed after the illuminance has stabilized, and main light shutdown and corridor light offline cannot occur simultaneously. Propagation analysis showed that reducing the main light brightness to 15% could reduce the illuminance in the passageway to 52 lx, exceeding the cross-space influence boundary; therefore, the minimum allowable brightness of the main light was revised to 25%. The final generated linkage safety envelope can continuously adjust with illuminance feedback, personnel movement, and equipment online status, providing dynamic, temporal, and verifiable safety constraints for comparing candidate linkage strategies, thereby reducing the risk of equipment conflicts and environmental state out-of-bounds violations.

[0072] In some embodiments, home appliances are controlled according to an executable linkage contract diagram, generating a linkage execution trajectory, including: Read the action nodes that meet the execution conditions in the executable linkage contract diagram, and determine the execution order of the device control actions according to the action dependency relationship; Convert device control actions into control commands that match the target home appliances, and verify the corresponding device status and environmental status before execution; Send control commands to the target home appliances that have passed the review, and obtain the device's operating status and corresponding environmental response; When the device's operating status does not meet the execution conditions, an alternative action or a rollback action is executed according to the abnormal compensation relationship. Generate a coordinated execution trajectory by associating the device control actions, device action status, and environmental response according to the execution sequence.

[0073] Specifically, the executable linkage contract diagram is parsed and scheduled by the home edge control node. Each action node in the diagram records the target device identifier, control parameters, planned execution time, maximum waiting time, completion criteria, and exception handling entry point. Action dependency edges record relationships such as sequential execution, parallel execution, state triggering, and mutually exclusive execution. The edge control node establishes an action node state machine, dividing action nodes into states such as pending activation, pending review, executing, execution completed, execution exception, and compensated, and periodically scans the executable linkage contract diagram according to device feedback and environmental perception results. Only when the preceding action is completed, the synchronization condition is met, and there are no mutually exclusive actions occupying the space, does the corresponding action node enter the pending review state, avoiding concurrent issuance of conflicting instructions.

[0074] When determining the execution order, the edge control node first performs topological sorting of action dependencies to form a basic execution sequence, and then dynamically adjusts it based on state triggering conditions and device response latency. Actions that can be parallelized but affect different objects are assigned to the same execution batch; actions that affect the same environmental state and may overlap excessively are arranged in the order of primary adjustment actions being executed first, followed by compensation actions; for actions with completion dependencies, subsequent actions are released only after the preceding action meets the completion criterion. If the preceding action is not completed within the maximum waiting time, the relevant subsequent nodes are paused and the process switches to the exception compensation branch.

[0075] Device control actions must be converted into control commands recognizable by the target device after protocol adaptation. The home edge control node reads the communication protocol, function identifier, parameter encoding method, control precision, and confirmation mechanism based on the device registration information, mapping the unified action description into on / off control, mode switching, target value setting, or tiered adjustment commands. For devices that only support discrete settings, continuous target parameters are mapped to the nearest available setting that does not exceed the safety boundary; for devices that support gradual control, a single large adjustment can be broken down into multiple progressive commands. Each control command is associated with a transaction identifier, action node identifier, transmission sequence number, and validity period to identify duplicate feedback and abnormal retransmissions.

[0076] Pre-execution review confirms that the current state still meets the execution conditions of the action node. The edge control node reads the target device's online status, current mode, protection status, manual lock status, and most recent control results, and obtains the personnel occupancy, environmental parameters, and status credibility of the corresponding space. If the device has been manually adjusted by the user, personnel status has changed, or the environmental status has reached the target range, the current action is canceled, downgraded, or recalculated; if the perceived credibility is insufficient, high-impact actions are delayed, and only low-risk actions are allowed to be executed. After the review is passed, the system locks the relevant device resources and sends control commands to prevent the same device from receiving conflicting controls before the action is completed.

[0077] Furthermore, after the command is issued, the edge control node synchronously collects device confirmation feedback, operating parameters, and environmental sensor data. The device's action status is determined based on command reception confirmation, execution progress, target parameter attainment, and completion feedback. The environmental response is determined based on the amount, direction, and response delay of changes in status nodes within its effective range. The system compares the device feedback with completion criteria; for example, a curtain reaching 95% or more of its position is considered a complete closure, and a light fixture's brightness entering the allowable deviation range and maintaining it for 2 seconds is considered a complete dimming. If the device feedback is normal but the environment does not produce the expected response, the action is marked as a response deviation, rather than directly determining device failure.

[0078] When a device's action status does not meet the execution conditions or completion criteria, the system enters the corresponding compensation relationship based on the exception type. Communication timeouts can be resent within a limited number of attempts. When a device is offline, a replacement device with the same target and similar scope of action is selected. If a device is stuck or its parameters exceed limits, subsequent dependent actions are stopped and a rollback action is executed. Rollback actions can restore the state of previous devices, maintain a safe intermediate state, or release occupied resources. A status check is also performed before compensation execution, and compensation actions have higher priority than ordinary pending actions.

[0079] For example, in a living room movie-watching scenario, the system schedules closing the curtains and adjusting the main light to 25% as the first batch of actions. Turning on the hallway light is triggered when the illuminance in the passageway area drops below 75 lx. Starting the audio-visual equipment is triggered after the illuminance in the sofa area reaches 80-150 lx and user A is still in the sofa area. Eight seconds after the curtain command is issued, only feedback indicates the curtain has moved to 60% of its original position, exceeding the completion time limit. The system then pauses the audio-visual equipment startup, performs a safety fallback by adjusting the main light to 35%, and calls the zoned blackout device as an alternative action. Subsequently, it detects that the illuminance in the sofa area stabilizes at 132 lx and the illuminance in the passageway area is 66 lx. The system then releases the audio-visual equipment startup node and completes playback control.

[0080] During execution, edge control nodes, following a unified clock, associate the activation time of action nodes, control commands, device confirmation status, actual completion status, anomaly type, compensation actions, and environmental responses to form a coordinated execution trajectory. Each time segment in the trajectory is associated with a device object, spatial object, action result, and status reliability, comprehensively reflecting the actual process from action scheduling to environmental response. This provides a data foundation for subsequent deviation analysis, deviation source location, and local repair, and improves the controllability and fault tolerance of multi-device coordinated execution.

[0081] In some embodiments, based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory, the affected equipment control actions are partially repaired or the target linkage strategy is redefined, including: The linkage execution trajectory and the state change trajectory are aligned in time to determine the equipment action deviation and state response deviation. Track state response deviations along the dynamic interaction relationships in the home status twin diagram to determine the source and scope of the deviations; When the propagation range meets the local repair conditions, freeze the device control actions outside the propagation range, and correct the affected device control actions within the feasible domain defined by the linkage control target. When the propagation range does not meet the local repair conditions or the correction result does not meet the linkage safety envelope, the target linkage strategy is redefined based on the current home status.

[0082] Specifically, the home edge control node saves the state change trajectory corresponding to the actions of each candidate device before the target linkage strategy is executed, and continuously writes the linkage execution trajectory during the execution process. The state change trajectory records the predicted action start time, device completion time, device parameter changes, and environmental state response, while the linkage execution trajectory records the actual command issuance, device confirmation, action completion, compensation processing, and environmental feedback. To avoid direct comparison distortion caused by differences in communication latency, sensor sampling period, and device response speed, the system establishes a correspondence between action node identifiers, device identifiers, and state node identifiers, and uses the action activation time as the local time origin to perform segmented temporal alignment on the two types of trajectories.

[0083] Timing alignment can be achieved using a combination of event anchors and dynamic time windows. Edge control nodes first use common events such as command issuance, device startup, and target parameter arrival as anchors. Then, between adjacent anchors, the predicted and actual sequences are resampled according to the device's allowable response delay. If the actual action is later than the predicted action but still within the allowable delay range, only the delay deviation is recorded; if the device does not start, terminates prematurely, or the executed parameters differ from the planned values, a device action deviation is generated. For environmental conditions such as illuminance, temperature, and noise, the system compares the actual response with the predicted response in terms of direction, magnitude, rate of change, and settling time to generate state response deviations. Each deviation is associated with its occurrence time, deviation amount, duration, and confidence level, forming a deviation event sequence.

[0084] When determining the source of the deviation, the node exhibiting the state response deviation is used as the starting point for tracing, and a reverse search is performed along the device action edges, environmental propagation edges, spatial adjacency edges, and device coupling edges in the home state twin graph. The system determines whether the upstream device action can explain the current deviation based on the edge's action direction, intensity, response delay, and reliability, and matches the actual deviation arrival time with the expected response delay of the associated edge. If a device action deviation is consistent with the downstream state response deviation in direction, timing, and contribution, the corresponding device node is identified as the source of the deviation. Subsequently, the deviation impact propagates along the positive dynamic action relationship, estimating the deviation increment of each downstream node until the impact falls below the propagation threshold, exceeds the target space, or is blocked by a stable state node, thus determining the propagation range.

[0085] Local repair conditions can be determined based on the number of affected nodes, propagation level, proportion of critical states, and remaining response time. When the deviation involves only a small number of devices and adjacent environmental nodes, does not affect critical safety states, and the remaining device capacity is sufficient to compensate, the propagation range is deemed to meet the local repair conditions. The system freezes device control actions completed or stably executing outside the propagation range, maintaining the corresponding action parameters and dependencies unchanged. Only device nodes, environmental nodes, and alternative actions within the propagation range are extracted to construct a local repair subgraph. Freezing prevents unrelated devices from undergoing repeated adjustments due to local anomalies and reduces the scale of re-solution.

[0086] The feasible domain for the linkage control target is comprised of the target state range, completion time limit, equipment operating range, and action constraints. The system calculates the remaining target state changes based on the current deviation, selects actions from the equipment capability diagram that can compensate for the changes, and limits the action amplitude, start time, and duration. During local solution processing, the original action parameters are adjusted first, followed by calling alternative equipment with the same target, and verifying whether the corrected state trajectory enters the linkage safety envelope. If multiple feasible repair actions exist, they can be sorted according to compensation completion time, additional energy consumption, number of actions, and degree of interference to the user, selecting the repair scheme with lower cost and stable state convergence.

[0087] For example, during the movie-watching linkage execution, it was predicted that the curtains would close to 95% within 8 seconds, and the main light would stabilize at 120 lx after being adjusted to 25%. In actual execution, the curtains only closed to 60% due to track resistance, and the illuminance in the sofa area remained at 178 lx at 12 seconds, while the main light's movement and the hallway light's status both matched the prediction. After aligning the two types of trajectories, the system obtained a curtain position deviation of 35 percentage points and a sofa area illuminance deviation of 58 lx. Reverse tracing showed that the negative effect from the curtain node to the sofa area illuminance node did not reach the predicted intensity, and the deviation did not propagate to temperature, audio-visual equipment, or bedroom status; the propagation range only included the curtains, main light, and living room illuminance node, meeting the conditions for local repair.

[0088] The system freezes the actions of corridor lights and audio-visual equipment, recalculating the main light adjustment within the target viewing range of 80-150 lx. Calculations show that reducing the main light from 25% to 18% will lower the illuminance to 138 lx after 5 seconds, while the illuminance in the passageway remains above 60 lx. The correction trajectory meets the linkage safety envelope, therefore local dimming repair is performed. If the curtain malfunction simultaneously causes changes in external lighting in the bedroom, a drop in illuminance in the passageway below the lower limit, or if available lighting actions cannot compensate within the safety boundary, then the propagation range does not meet the local repair conditions. The system updates the home status twin map with the latest personnel locations, equipment status, and environmental status, regenerating the linkage control target, candidate linkage strategies, and target linkage strategy.

[0089] Through the above processing, actual execution deviations can be refined from simple result anomalies into equipment action deviations and state response deviations, and the source and propagation range of deviations can be located based on dynamic interaction relationships. For local anomalies, only the affected actions are corrected, while for widespread anomalies, the target linkage strategy is redefined. This reduces redundant control of unrelated equipment, prevents the continuous spread of deviations, and improves the real-time performance, stability, and recovery capability of linkage control.

[0090] The following are system embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application.

[0091] Figure 2 This is a schematic diagram of the structure of a smart home device linkage control system based on multimodal perception provided in an embodiment of this application. Figure 2 As shown, the system includes: The acquisition module 201 is used to acquire multimodal sensing data and device operation data in the home space, perform time-series correction, object association and credibility assessment on the multimodal sensing data, and generate human settlement events. Update module 202 is used to update the home status twin based on human settlement events. The home status twin represents the dynamic interaction between users, home space, environmental status and home equipment. Extraction module 203 is used to extract intent evidence subgraphs from the home status twin graph, determine user intent and scene transition status based on intent evidence subgraphs, and generate linkage control targets; The deduction module 204 is used to generate candidate linkage strategies based on the linkage control target and equipment capability diagram, and to deduce the state change trajectory of the candidate linkage strategies using the home status twin diagram. The comparison module 205 is used to compare the state change trajectory with the linkage safety envelope, determine the target linkage strategy based on the comparison result, and convert the target linkage strategy into an executable linkage contract diagram containing action dependencies, execution conditions and exception compensation relationships. The generation module 206 is used to control home appliances according to the executable linkage contract diagram and generate linkage execution trajectory; Repair module 207 is used to perform local repairs or redetermine the target linkage strategy for the affected equipment control actions based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory.

[0092] In some embodiments, Figure 2The acquisition module 201, based on a unified time benchmark and a home space semantic benchmark, performs time alignment and spatial mapping on multimodal perception data to generate modal events representing state changes; determines perception credibility based on the data quality, perception continuity, and spatial matching degree of each modal event; performs cross-modal association on modal events and device operation data according to user objects, spatial objects, and associated time windows to generate candidate residential events; determines modal conflict degree based on the spatiotemporal consistency, perception credibility, and state continuity of each modal event in the candidate residential events; and merges candidate residential events or retains candidate states based on the modal conflict degree to generate residential events.

[0093] In some embodiments, Figure 2 The update module 202 maps the event state to the corresponding state node in the home state twin graph according to the user object, space object and device object associated with the human settlement event; updates the current state, change trend and state credibility of the corresponding state node according to the state change, perceived credibility and occurrence sequence of the human settlement event; determines the state association edge associated with the update node based on the interaction relationship in the home space, and corrects the action direction, action strength and response delay of the state association edge according to the state difference before and after the event; propagates the state change along the corrected state association edge, updates the affected state node, and forms a home state twin graph representing the current home state and change trend according to the propagation result.

[0094] In some embodiments, Figure 2 The extraction module 203 extracts the state change path along the dynamic interaction relationship based on the state nodes associated with the current residential event, and constructs an intent evidence subgraph; based on the intent evidence from different sources and their corresponding credibility in the intent evidence subgraph, it determines the evidence support of candidate user intent; combined with the temporal evolution of candidate user intent and its matching relationship with historical scene migration patterns, it determines user intent, scene migration state, and intent uncertainty; based on user intent, scene migration state, and intent uncertainty, it performs a difference mapping between the current home state and the target scene state, and generates a linkage control target that limits the target state range, response constraints, and device action range.

[0095] In some embodiments, Figure 2 The deduction module 204 decomposes the linkage control target into target state changes, matches candidate device actions that can drive the target state changes along the device action relationship in the device capability diagram; determines the action adaptability based on the action range, response delay and action constraints of the candidate device actions, and combines the candidate device actions based on the dependency and coupling relationships between devices to generate candidate linkage strategies; maps the device actions corresponding to each candidate linkage strategy to the home state twin diagram, propagates the action influence according to the dynamic action relationship, and predicts the temporal changes of related state nodes; and aggregates the device state changes and environmental responses to generate the state change trajectory corresponding to each candidate linkage strategy.

[0096] In some embodiments, Figure 2 The comparison module 205 matches the trajectory of each state change with the state boundary, rate of change constraint, and device interlock constraint in the linkage safety envelope according to the state node and evolution sequence to determine the trajectory out-of-bounds risk; it filters out candidate linkage strategies that do not meet the safety conditions due to trajectory out-of-bounds risk, and determines the target linkage strategy based on the goal achievement degree, response cost, and action stability of the remaining candidate linkage strategies; it generates action nodes and action dependencies according to the action sequence and state dependency relationship of each device in the target linkage strategy; it configures the execution conditions based on the device state and environment state for each action node, and establishes anomaly compensation relationship based on the corresponding alternative action and fallback path according to the action failure or state deviation, and generates an executable linkage contract diagram.

[0097] In some embodiments, Figure 2 The comparison module 205 extracts the state nodes and dynamic interaction relationships associated with the linkage control target from the home status twin graph; based on the current state, change trend and state credibility of the state nodes, combined with the action constraints in the equipment capability graph, it determines the allowable state range and allowable change rate; it propagates the expected impact of the equipment actions corresponding to the candidate linkage strategy along the dynamic interaction relationship, and determines the equipment interlock boundary and cross-space influence boundary; it associates the allowable state range, allowable change rate, equipment interlock boundary and cross-space influence boundary according to the evolution time sequence, and generates a linkage safety envelope that is dynamically updated with the current home status.

[0098] In some embodiments, Figure 2 The generation module 206 reads the action nodes that meet the execution conditions in the executable linkage contract diagram, determines the execution order of the device control actions according to the action dependency relationship, converts the device control actions into control instructions that match the target home appliances, and reviews the corresponding device status and environmental status before execution; sends control instructions to the target home appliances that have passed the review, and obtains the device action status and the corresponding environmental response; when the device action status does not meet the execution conditions, it executes alternative actions or rollback actions according to the abnormal compensation relationship; and generates a linkage execution trajectory by associating the device control actions, device action status and environmental response according to the execution sequence.

[0099] In some embodiments, Figure 2The repair module 207 performs time-series alignment between the linkage execution trajectory and the state change trajectory to determine the device action deviation and state response deviation; it tracks the state response deviation along the dynamic interaction relationship in the home state twin diagram to determine the source and propagation range of the deviation; when the propagation range meets the local repair conditions, it freezes the device control actions outside the propagation range and corrects the affected device control actions within the feasible domain defined by the linkage control target; when the propagation range does not meet the local repair conditions or the correction result does not meet the linkage safety envelope, it redetermines the target linkage strategy based on the current home state.

[0100] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0101] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various system embodiments described above.

[0102] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0103] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0104] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0106] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0107] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A smart home device linkage control method based on multi-modal perception, characterized in that, include: Acquire multimodal sensing data and device operation data within the home space, perform time-series correction, object association, and credibility assessment on the multimodal sensing data, and generate human settlement events; The home status twin is updated based on the aforementioned human settlement events. The home status twin represents the dynamic interaction between users, home space, environmental status, and home appliances. Extract the intent evidence sub-graph from the home status twin graph, determine the user intent and scene transition status based on the intent evidence sub-graph, and generate linkage control target; Candidate linkage strategies are generated based on the linkage control target and equipment capability diagram, and the state change trajectory of the candidate linkage strategies is deduced using the home status twin diagram; The state change trajectory is compared with the linkage safety envelope, the target linkage strategy is determined based on the comparison result, and the target linkage strategy is converted into an executable linkage contract graph containing action dependencies, execution conditions and anomaly compensation relationships. Control home appliances according to the executable linkage contract diagram and generate linkage execution trajectory; Based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory, the affected equipment control actions are partially repaired or the target linkage strategy is redefined.

2. The method according to claim 1, characterized in that, The process of acquiring multimodal sensing data and device operation data within the home space, performing time-series correction, object association, and credibility assessment on the multimodal sensing data, and generating human settlement events includes: Based on a unified time reference and a home space semantic reference, the multimodal perception data is time-aligned and spatially mapped to generate modal events that represent state changes. The perception credibility is determined based on the data quality, perception continuity, and spatial matching degree of each modal event; Based on user objects, spatial objects, and associated time windows, cross-modal associations are performed on the modal events and device operation data to generate candidate habitation events; The modal conflict degree is determined based on the spatiotemporal consistency, perceived credibility, and state continuity of each modal event in the candidate habitation event. Based on the modal conflict degree, candidate habitation events are fused or candidate states are retained to generate a habitation event.

3. The method according to claim 1, characterized in that, The updating of the home status twin map based on the residential events includes: Based on the user objects, space objects, and device objects associated with the aforementioned residential events, the event states are mapped to the corresponding state nodes in the residential state twin graph; Based on the state changes, perceived credibility, and occurrence sequence of the human settlement events, update the current state, change trend, and state credibility of the corresponding state nodes; Based on the interaction relationships within the home space, determine the state association edges associated with the update nodes, and correct the direction, intensity, and response delay of the state association edges according to the state differences before and after the event. Propagate state changes along the corrected state-related edges, update affected state nodes, and form a home status twin graph representing the current home status and its changing trends based on the propagation results.

4. The method according to claim 1, characterized in that, The step of extracting an intent evidence subgraph from the home status twin graph, determining the user's intent and scene transition status based on the intent evidence subgraph, and generating a linkage control target includes: Based on the state nodes associated with the current human settlement event, extract the state change path along the dynamic interaction relationship and construct the intention evidence subgraph; Based on the intent evidence from different sources and their corresponding credibility in the intent evidence subgraph, the evidence support level of the candidate user's intent is determined. By combining the temporal evolution of the candidate user intents with their matching relationship with historical scene migration patterns, the user intent, scene migration status, and intent uncertainty are determined. Based on the user intent, scene migration status, and intent uncertainty, the current home status is mapped to the target scene status to generate a linkage control target that limits the target state range, response constraints, and device scope.

5. The method according to claim 1, characterized in that, The step of generating candidate linkage strategies based on the linkage control target and equipment capability diagram, and using the home status twin diagram to deduce the state change trajectory of the candidate linkage strategies, includes: The linkage control target is decomposed into target state changes, and candidate device actions that can drive the target state changes are matched along the device action relationship in the device capability diagram. The action adaptability is determined based on the scope of action, response latency and action constraints of the candidate device actions, and the candidate device actions are combined based on the dependencies and coupling relationships between devices to generate candidate linkage strategies; The device actions corresponding to each candidate linkage strategy are mapped to the home status twin graph, and the action influence is propagated according to the dynamic interaction relationship to predict the temporal changes of related status nodes. By aggregating device status changes and environmental responses, the system generates status change trajectories corresponding to each candidate linkage strategy.

6. The method according to claim 1, characterized in that, The step of comparing the state change trajectory with the linkage safety envelope, determining the target linkage strategy based on the comparison result, and converting the target linkage strategy into an executable linkage contract graph containing action dependencies, execution conditions, and anomaly compensation relationships includes: According to the state nodes and evolution sequence, the trajectory of each state change is matched with the state boundary, change rate constraint and device interlock constraint in the linked safety envelope to determine the trajectory out-of-bounds risk; Candidate linkage strategies that do not meet the safety conditions and have the risk of trajectory going out of bounds are screened out. The target linkage strategy is determined based on the degree of goal achievement, response cost and action stability of the remaining candidate linkage strategies. According to the action sequence and state dependency of each device action in the target linkage strategy, generate action nodes and action dependencies; Configure execution conditions for each action node based on device and environment status, and establish anomaly compensation relationships based on the corresponding alternative actions and fallback paths for action failures or state deviations, generating an executable linkage contract diagram.

7. The method according to claim 6, characterized in that, The linked security envelope is generated in the following manner: Extract the state nodes and dynamic interaction relationships associated with the linkage control target from the home status twin graph; Based on the current state, change trend, and state reliability of the state node, and in conjunction with the action constraints in the device capability diagram, the allowable state range and allowable change rate are determined. The expected impact of the device actions corresponding to the candidate linkage strategy is propagated along the dynamic interaction relationship to determine the device interlock boundary and cross-space impact boundary; The allowed state range, allowed rate of change, device interlock boundary, and cross-space influence boundary are associated according to the evolution sequence to generate a linkage safety envelope that is dynamically updated with the current home status.

8. The method according to claim 1, characterized in that, The step of controlling home appliances according to the executable linkage contract diagram and generating a linkage execution trajectory includes: Read the action nodes that meet the execution conditions in the executable linkage contract diagram, and determine the execution order of the device control actions according to the action dependency relationship; The device control actions are converted into control commands that match the target home appliances, and the corresponding device status and environmental status are verified before execution; Send control commands to the target home appliances that have passed the review, and obtain the device's operating status and corresponding environmental response; When the device's operating state does not meet the execution conditions, an alternative action or a rollback action is executed according to the aforementioned abnormal compensation relationship; Generate a coordinated execution trajectory by associating the device control actions, device action status, and environmental response according to the execution sequence.

9. The method according to claim 1, characterized in that, The step of performing local repairs or redetermining the target linkage strategy for the affected equipment control actions based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory includes: The linkage execution trajectory and the state change trajectory are time-aligned to determine the equipment action deviation and state response deviation. Track the state response deviation along the dynamic interaction relationship in the home status twin diagram to determine the source and propagation range of the deviation; When the propagation range meets the local repair conditions, freeze the device control actions outside the propagation range, and correct the affected device control actions within the feasible domain defined by the linkage control target. When the propagation range does not meet the local repair conditions or the correction result does not meet the linkage safety envelope, the target linkage strategy is re-determined based on the current home status.

10. A smart home device linkage control system based on multimodal perception, characterized in that, include: The acquisition module is used to acquire multimodal sensing data and device operation data in the home space, perform time-series correction, object association and credibility assessment on the multimodal sensing data, and generate human settlement events; The update module is used to update the home status twin map based on the human settlement events. The home status twin map represents the dynamic interaction between users, home space, environmental status and home appliances. The extraction module is used to extract the intent evidence sub-graph from the home status twin graph, determine the user intent and scene transition status based on the intent evidence sub-graph, and generate linkage control targets. The deduction module is used to generate candidate linkage strategies based on the linkage control target and equipment capability diagram, and to deduce the state change trajectory of the candidate linkage strategies using the home status twin diagram; The comparison module is used to compare the state change trajectory with the linkage safety envelope, determine the target linkage strategy based on the comparison result, and convert the target linkage strategy into an executable linkage contract graph that includes action dependencies, execution conditions, and anomaly compensation relationships. The generation module is used to control home appliances according to the executable linkage contract diagram and generate linkage execution trajectory; The repair module is used to perform local repairs or redetermine the target linkage strategy for the affected equipment control actions based on the deviation and propagation range between the linkage execution trajectory and the state change trajectory.

Images