A multi-modal data interaction display system of a smart home central control screen

Abstract

The application relates to the technical field of smart home, and discloses a multi-modal data interaction display system of a smart home central control screen, which comprises a state modeling module, an interaction analysis module, a decision optimization module, a state prediction module and an execution display module; the method comprises the following steps: firstly, modeling the environment state and analyzing multi-modal interaction to understand user intention and interaction mode; then, solving an optimal target state through an optimization model dynamically adjusted by the interaction mode and predicting a future state; finally, issuing an instruction and displaying visualized information of the current target and the future prediction on the central control screen to improve interaction intelligence. The application perceives interaction to dynamically make decisions, solves the problem of rigid control, provides active service by predicting the future state, overcomes the deficiency of passive response, displays current and future information on the screen, solves the defect of single display, and improves intelligence and interaction experience.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, specifically to a multimodal data interaction display system for a smart home central control screen. Background Technology

[0002] With the rapid development of IoT and AI technologies, smart home systems have gradually become widespread, serving as an important technological means to improve quality of life. Current smart home systems mostly achieve remote control and status monitoring of various devices in the home through central control screens, mobile applications, or voice assistants.

[0003] However, existing technologies still fall short in achieving deeper levels of intelligent and human-centered interaction. Current smart home systems largely rely on the precise parsing of direct user commands, but this interaction often remains at the semantic level, lacking the perception and utilization of deeper information such as the emotions and intent implied in the user's interaction. Their control strategies are typically rigid, passively responding to or executing preset fixed rules, making it difficult to dynamically adjust the focus of decisions based on real-time interaction scenarios, such as intelligently balancing user needs with system energy efficiency.

[0004] Furthermore, existing systems generally lack foresight and cannot predict future changes in the home environment, thus failing to provide proactive services or early warnings. The control screen's display interface is also often a simple list of current device statuses, presenting limited information and failing to provide users with insights into future environmental trends, thereby restricting the depth and efficiency of the human-computer interaction experience.

[0005] Therefore, this invention proposes a multimodal data interaction display system for smart home central control screens to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal data interaction display system for smart home control screens, which solves the problems of rigid interactive decision-making, lack of proactive services, and limited information display in smart home systems.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multimodal data interaction display system for a smart home central control screen, the system comprising:

[0008] The state modeling module is used to acquire device data of smart home devices, generate a current state vector representing the current environment based on the device data, and record the current state vector to form state history trajectory data.

[0009] The interaction parsing module is used to generate intent transformation operators that represent user intents and interaction feature vectors that represent interaction attributes based on multimodal interaction inputs.

[0010] The decision optimization module is used to combine the current state vector with the intention transformation operator and solve for the optimal target state vector through a constraint optimization model dynamically adjusted by the interaction feature vector.

[0011] The state prediction module is used to predict and generate future state vectors based on the historical trajectory data of the states using a time-series prediction model.

[0012] The execution display module is used to generate and issue device control commands based on the optimal target state vector, and to integrate and display the state visualization interface and predictive visual indicators corresponding to the optimal target state vector and the future state vector on the central control screen.

[0013] Preferably, the state modeling module includes:

[0014] The device data is subjected to a preset weighted aggregation operation to obtain multiple state components, and the multiple state components are combined into the current state vector; wherein, the state components are used to characterize the macroscopic dimension of the home environment.

[0015] Preferably, the interactive parsing module includes:

[0016] Based on the multimodal interactive input, speech semantic vectors and spatial semantic vectors are parsed and generated. The speech semantic vectors and spatial semantic vectors are then calculated using a fusion generation function to generate the intent transformation operator. Based on the non-semantic physical features in the multimodal interactive input, the interactive feature vector is extracted and generated.

[0017] Preferably, the decision optimization module includes:

[0018] The intention transformation operator is applied to the current state vector to generate a preliminary target state vector, and the preliminary target state vector is used as the optimization objective of the constrained optimization model. The optimal target state vector is generated by solving the constrained optimization model.

[0019] Preferably, the constrained optimization model includes an objective function, which is configured to minimize a weighted cost function;

[0020] The weighted cost function is calculated using the initial target state vector as input, and the weighted cost function includes:

[0021] The deviation cost item is calculated based on the deviation between the optimal target state vector and the preliminary target state vector, and the system operation cost item is calculated based on the optimal target state vector;

[0022] The decision optimization module dynamically adjusts the weight coefficients of each cost item in the weighted cost function based on the interaction feature vector.

[0023] Preferably, the constrained optimization model in the decision optimization module further includes constraints, which include:

[0024] Equipment physical constraints are used to ensure that the equipment parameters corresponding to the optimal target state vector are within a preset safe operating range;

[0025] Scene mode consistency constraints are used to ensure that the optimal target state vector meets the preset requirements of the current scene mode of the system.

[0026] Preferably, the state prediction module includes:

[0027] The future state vector is compared with a preset user preference model to determine the prediction deviation; and when the prediction deviation meets a preset trigger condition, an active interaction proposal is generated.

[0028] Preferably, the execution display module is further configured to receive the proactive interaction proposal and display the proactive interaction proposal on the central control screen.

[0029] Preferably, the execution display module generates the device control command as follows:

[0030] The optimal target state vector is used as input to calculate the device control command corresponding to the optimal target state vector through a preset reverse mapping function.

[0031] This invention also provides a multimodal data interaction display method for a smart home central control screen, the method comprising the following steps:

[0032] S1. Obtain device data of smart home devices, generate a current state vector representing the current environment based on the device data, and record the current state vector to form state history trajectory data;

[0033] S2. Based on multimodal interactive input, generate an intent transformation operator representing the user's intent and an interactive feature vector representing the interactive attributes.

[0034] S3. Based on the current state vector, the intention transformation operator, and the interaction feature vector, the optimal target state vector is generated by solving a constraint optimization model dynamically adjusted by the interaction feature vector.

[0035] S4. Based on the historical trajectory data of the state, predict and generate future state vectors using a time-series prediction model;

[0036] S5. Generate and issue device control commands based on the optimal target state vector, and display the state visualization interface and predictive visual indication corresponding to the optimal target state vector and the future state vector respectively on the central control screen.

[0037] This invention provides a multimodal data interaction display system for a smart home central control screen. It has the following beneficial effects:

[0038] 1. This invention employs a decision optimization scheme that combines intent transformation operators and interaction feature vectors. The system solves the problem through a constraint optimization model dynamically adjusted by the interaction feature vectors; it achieves a deep understanding and flexible execution of user intent; compared to existing technologies that rely on fixed rules or simple instruction parsing, it solves the shortcomings of rigid control strategies and the inability to perceive user interaction states, making home control more intelligent and user-friendly.

[0039] 2. This invention achieves accurate prediction of the future state of the home environment by establishing a state history trajectory and using a time-series prediction model. The system can compare the prediction results with the user preference model, thereby proactively generating interactive proposals. Compared with the passive control of existing technologies that simply respond to user commands, this invention solves the defects of lacking foresight and being unable to provide proactive services, thereby improving the system's intelligence and user experience.

[0040] 3. This invention provides a fusion display method that combines a visual interface of the optimal target state with a predictive visual indication of the future state on a central control screen. As a result, users can simultaneously see the immediate results of the operation and the future trend of the environment. Compared with the single information interface in the prior art that can only display the current state of the device, this invention solves the shortcomings of limited display content and lack of predictive information, effectively reduces the user's uncertainty about the future state of the environment, and enhances the depth and effectiveness of human-computer interaction. Attached Figure Description

[0041] Figure 1 This is a system architecture diagram of the present invention;

[0042] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Please see Figure 1This invention provides a multimodal data interaction display system for a smart home central control screen, the system comprising:

[0045] The state modeling module is used to acquire device data of smart home devices, generate a current state vector representing the current environment based on the device data, and record the current state vector to form state history trajectory data.

[0046] In this embodiment, the state modeling module is responsible for transforming the scattered and heterogeneous device data within the smart home system into a unified, structured current state vector that can macroscopically represent the overall environmental state.

[0047] Specifically, the state modeling module continuously acquires real-time device data from various smart home devices deployed in the system. This device data consists of raw sensor readings or state parameters describing the physical state of the devices themselves. For example, the device data may include, but is not limited to:

[0048] Temperature sensors provide Celsius readings, humidity sensors provide relative humidity percentages, smart lights provide luminance lumen values ​​and color temperature Kelvin values, air quality monitors provide PM2.5 or carbon dioxide concentration values, and smart curtains provide opening / closing percentages, etc. This data comes from different manufacturers and follows different protocols; its raw form is difficult to use directly for comprehensive environmental decision-making.

[0049] To address this issue, this invention introduces the concept of macroscopic dimensions of the home environment and maps raw device data onto these dimensions, forming multiple state components. Each state component is a quantitative representation of a specific macroscopic dimension of the home environment at the current moment. In one possible implementation, the macroscopic dimensions may include thermal comfort, lighting environment, air quality, and spatial security dimensions.

[0050] The state modeling module calculates each state component by performing a preset weighted aggregation operation. This operation takes multiple related device data as input and aggregates them to generate a single scalar value as the state component. The calculation formula for the weighted aggregation operation can be expressed as:

[0051]

[0052] In the formula: S k,t This represents the k-th state component calculated at time t, such as the thermal comfort component; k is the index of the state component; i is the device index associated with the k-th state component; N k d represents the total number of devices associated with calculating the k-th state component; k,i,tThis represents the raw device data reported by the i-th device at time t, which is related to the k-th state component, such as the temperature reading of a specific temperature and humidity sensor; w k,i This represents a preset weighting coefficient used to adjust the importance of the device data of the i-th device in calculating the k-th state component.

[0053] The weighting coefficient w k,i The settings are crucial for achieving accurate state modeling. These weighting coefficients can be pre-configured in the system based on factors such as the physical location of the devices, the importance of the space, the accuracy of the sensors, or user-defined preferences. For example, when calculating the thermal comfort component of a whole house, the weighting coefficients of temperature sensors located in the living room or bedroom are typically higher than those of temperature sensors located in the storage room or hallway.

[0054] After calculating all state components (e.g., S) 1,t S 2,t After that, the state modeling module combines these state components into a column vector, forming the current state vector at time t. This vector provides other modules of the system with a comprehensive, quantitative, and easily manageable view of the current home environment.

[0055] Furthermore, the state modeling module also generates the current state vector at each time step. The data is recorded and stored. By accumulating these vectors in chronological order, the system forms a historical trajectory data of the state. This historical trajectory data fully captures the dynamic pattern of the home environment's state evolving over time, providing the necessary data foundation for the subsequent state prediction module to perform time-series predictions, thereby supporting the system's forward-looking functions.

[0056] The interaction parsing module is used to generate intent transformation operators that represent user intents and interaction feature vectors that represent interaction attributes based on multimodal interaction inputs.

[0057] In this embodiment, the function of the interaction parsing module is to deeply parse the multimodal interaction input issued by the user and transform it into a set of structured data required for system decision-making. Specifically, it includes an intent transformation operator that represents the core intent of the user's command, and an interaction feature vector that represents the user's interaction mode or emotional state.

[0058] In one possible implementation, the multimodal interactive input is a combination of data streams generated by the user interacting with the system through multiple channels. For example, the user can touch or swipe on the central control screen with their finger while simultaneously interacting with voice commands. The interaction parsing module is configured to synchronously receive and process these different modal input data.

[0059] The process of generating the intent transformation operator involves parsing and quantifying the semantic content of the user's command. Specifically, when a multimodal interactive input is received, the interaction parsing module will parse out the speech semantic vector and the spatial semantic vector in parallel.

[0060] The speech semantic vector is a vectorized representation of the action or intention to change contained in a speech command. This process may include: converting the user's speech signal into a text string using speech recognition technology, then analyzing the text using a natural language understanding model to extract the core control intention. For example, for the voice command "Turn this brighter," the system can extract the core intention of "increase brightness" and encode it into a predefined speech semantic vector.

[0061] The spatial semantic vector is a vectorized representation of the spatial range or object targeted by the user's interaction. This vector is typically parsed from non-voice interaction modalities. For example, if a user issues the aforementioned voice command and touches the area representing the "living room" on the central control screen, the system maps the screen coordinates of that touch to the logical space of "living room" and encodes it as a spatial semantic vector. The purpose of this vector is to limit the scope of application of the user's intent.

[0062] After generating the speech semantic vector and spatial semantic vector, the interaction parsing module uses a fusion generation function to calculate the two to generate the final intent transformation operator T. intent The intent transformation operator defines the specific operations for transforming the current state vector into the target state vector desired by the user. The fusion generation function can be designed as follows:

[0063] Intended transformation operator generation formula:

[0064]

[0065] In the formula: The intention transformation operator acts on any state vector. Transformation operations on; The input is the current state vector; It is composed of speech semantic vectors The resulting state change vector specifies the components in the state vector that should change and the magnitude of their change. For example, "brighten up" corresponds to a positive increase in the ambient light component; P spatial It is composed of spatial semantic vectors The resulting spatial selection matrix is ​​typically a diagonal matrix. Its diagonal elements identify the state components affected by the user's intent. If the user intent affects a state component, the corresponding diagonal element is 1; otherwise, it is 0. This matrix ensures that state changes occur only in the macroscopic dimension specified by the user.

[0066] Simultaneously, the interaction parsing module also extracts and generates the interaction feature vector based on the non-semantic physical features in the multimodal interaction input. These non-semantic physical features refer to physical attributes that do not affect the semantics of the instruction itself but reflect the potential state of the user during interaction. In some embodiments, these features may include the volume of the user's speech, speech rate, pitch, and the force, speed, and smoothness of the user's touch or gesture.

[0067] The interactive parsing module extracts these original physical features, normalizes them, and then combines them into an interactive feature vector. This vector provides a quantitative basis for the user's current interaction state for the subsequent decision optimization module, enabling the system to dynamically adjust its decision strategy. For example, when the user's command is perceived to be urgent, priority is given to ensuring response speed rather than energy consumption economy.

[0068] The decision optimization module is used to combine the current state vector with the intention transformation operator and solve for the optimal target state vector through a constraint optimization model dynamically adjusted by the interaction feature vector.

[0069] In this embodiment, the core task of the decision optimization module is to, upon receiving the current state vector representing the environment and the transformation operator representing the user's intent, solve for and generate an optimal target state vector through a constraint optimization model that comprehensively considers multiple factors. This optimal target state vector satisfies the user's core needs while also taking into account the system's operational efficiency and security.

[0070] Generally, the decision optimization module obtains the current state vector from the state modeling module. And obtain the intent transformation operator T from the interaction parsing module. intent and interaction feature vector

[0071] In one possible implementation, the decision optimization module first applies the intention transformation operator to the current state vector to generate a preliminary target state vector. This initial target state vector is the most direct, unoptimized mathematical translation of the user command. Its generation process can be represented as:

[0072] Preliminary target state vector generation formula:

[0073]

[0074] In the formula: Let T be the initial target state vector; intent (·) represents the transformation function represented by the intended transformation operator; The current state vector is the input.

[0075] The initial target state vector While reflecting the user's direct intent, it may not be the optimal solution at the system level. Therefore, it will serve as a core optimization objective of the constrained optimization model, used to solve for the final optimal objective state vector.

[0076] Specifically, the constrained optimization model is constructed as a problem aimed at minimizing a weighted cost function. The weighted cost function J integrates cost considerations from multiple dimensions, and its expression can be:

[0077] Weighted cost function:

[0078]

[0079] In the formula: Let J be the optimal objective state vector to be solved; J is the total weighted cost; the first term... The deviation cost term is calculated by taking the square of the Euclidean distance between the optimal target state vector and the initial target state vector, and is used to penalize solutions that deviate from the user's original intent. λ represents the system operating cost, which is a function used to evaluate the system operating cost corresponding to the optimal target state vector. For example, this function can be designed as a linear combination of energy-related state components (such as illumination and temperature) to quantify the energy cost required for the system to reach that state; dev and λ op These are the weighting coefficients for the deviation cost item and the system operating cost item, respectively.

[0080] The weighting coefficient λ dev and λ op It is not a fixed value, but rather determined by the interaction feature vector. It is dynamically adjusted. The decision optimization module internally includes a weight adjustment function F. weights This function takes the interactive feature vector as input and outputs a set of weight coefficient values.

[0081] Weight coefficient dynamic adjustment function:

[0082]

[0083] For example, when the interaction feature vector When components in the speech pattern (such as high volume or fast speech rate) indicate that the user may be in a state of urgency or impatience, the weight adjustment function F... weights The output will be a higher λ. dev value and a lower λ op This causes the optimization model to prioritize minimizing the deviation from the user's intent during the solution process, meaning it tends to fulfill user instructions quickly and accurately, while its sensitivity to system operating costs is correspondingly reduced. Conversely, when interaction characteristics indicate that the user's state is stable, λ can be appropriately increased. op The weighting of the system is determined in order to find a more energy-efficient system operation solution while meeting user needs.

[0084] Furthermore, the solution process for the aforementioned optimization problem must also satisfy a set of pre-defined constraints to ensure that the final generated optimal target state vector is safe and suitable in the real world.

[0085] The constraints specifically include:

[0086] Equipment physical constraints ensure that the component values ​​of the optimal target state vector, when mapped back to the control parameters of a specific device, fall within the physically safe operating range of the device or a manufacturer-preset range. This constraint can be formally expressed as... in and These are vectors representing the lower and upper bounds of each dimension in the state space, respectively.

[0087] Scene mode consistency constraints are used to maintain the atmosphere and rules of the overall scene mode (such as cinema mode or sleep mode) in which the system is currently operating. When the system is in a specific scene mode, this constraint restricts the solution space of the optimal objective state vector, ensuring that it falls within the set of states allowed by that scene mode. For example, in "sleep mode," there is an upper limit to the allowed values ​​of ambient lighting components; even if the user requests higher brightness, the final solution will still be limited by this constraint.

[0088] By employing a numerical optimization solver to solve the aforementioned optimization problem with dynamic weights and multiple constraints, the decision optimization module ultimately obtains the optimal objective state vector. It is then output to subsequent modules for execution and display.

[0089] The state prediction module is used to predict and generate future state vectors based on the historical trajectory data of the states using a time-series prediction model.

[0090] In this embodiment, the state prediction module aims to give the smart home system a forward-looking capability, providing users with predictive visual instructions and supporting proactive interactive services by predicting the future evolution of the home environment.

[0091] Typically, the state prediction module receives historical state trajectory data continuously generated and recorded by the state modeling module. This data is a time series, consisting of multiple current state vectors arranged in chronological order, and can be represented as follows: in It is the current state vector at the most recent moment, and n is the length of the time window of the historical data used.

[0092] In one possible implementation, the state prediction module internally deploys a pre-trained temporal prediction model to learn and capture the complex dynamic patterns of state vector evolution over time. Alternatively, the temporal prediction model can be a deep learning model capable of effectively processing sequential data, such as a Long Short-Term Memory (LSTM) network or a Gated Recurrent Unit (GRU) network. This model takes the historical state trajectory data as input and, through its internal network structure, infers the environmental state after a predetermined time step h. This process can be represented by the following equation:

[0093] Future state vector prediction formula:

[0094]

[0095] In the formula: The future state vector predicted by the model at a future time t+h; This represents the pre-trained time series prediction model; The input is the historical trajectory data of the stated state.

[0096] A key function of the state prediction module is to provide proactive services using the prediction results. Specifically, this module further compares the predicted future state vector with a preset user preference model to identify potential user needs or environmental problems.

[0097] The user preference model is a data structure that represents the desired home environment state of a user under specific conditions. In some embodiments, this model can be a machine learning model obtained through long-term learning of the user's historical operations and environmental data, or it can be a set of rules preset by the user or the system. The model can provide a target state vector of user preference based on a future time point t+h or other contextual information.

[0098] The state prediction module calculates the future state vector. With the user preference state vector The difference between these values ​​is used to determine a prediction bias. This bias quantifies the degree of inconsistency between the natural evolutionary trend of the future environment and the user's expected state.

[0099] Prediction deviation calculation formula:

[0100]

[0101] In the formula: D dev The calculated prediction bias value is represented by distance(·,·); distance(·,·) represents a distance function used to measure the difference between two vectors, such as Euclidean distance or Mahalanobis distance. The future state vector generated for the prediction; This refers to the user preference state vector obtained from the user preference model at the same time.

[0102] After calculating the prediction deviation, the state prediction module compares the deviation value with a preset trigger condition. In one possible implementation, the trigger condition is a preset deviation threshold θ. dev When the prediction deviation satisfies D dev >θ dev This condition indicates that the system predicts that future environmental conditions will significantly deviate from the user's comfort zone or routine habits.

[0103] Once the triggering condition is met, the state prediction module will generate a clear and actionable proactive interaction proposal based on the specific content of the deviation. For example, if the deviation mainly comes from the air quality state component, and the system predicts that the carbon dioxide concentration will exceed the user's preferred threshold in the future, the generated proactive interaction proposal could be: "Indoor air may become stuffy; should we turn on ventilation for you in advance?"

[0104] The generated proactive interaction proposal is then sent to the execution display module for presentation to the user on the central control screen, thereby realizing a closed-loop, intelligent proactive user service.

[0105] The execution display module is used to generate and issue device control commands based on the optimal target state vector, and to integrate and display the state visualization interface and predictive visual indicators corresponding to the optimal target state vector and the future state vector on the central control screen.

[0106] In this embodiment, the execution display module serves as the final execution endpoint of the system decision and the core interface for human-computer interaction. It is responsible for transforming the abstract decisions generated by the upstream modules into physical world device actions and visual information on the screen. The main functions of this module include generating and issuing device control commands, as well as generating and presenting the integrated display interface.

[0107] Regarding the generation and issuance of equipment control commands, the execution display module receives the optimal target state vector calculated by the decision optimization module. This vector is an ideal description of the macroscopic state of the home environment; it is abstract data and cannot be directly used to control specific physical devices. Therefore, it needs to be converted into a set of explicit device control commands.

[0108] Specifically, the execution display module performs reverse analytical calculation on the optimal target state vector through a preset reverse mapping function. The design of this reverse mapping function corresponds to the weighted aggregation operation in the state modeling module, aiming to deconstruct macroscopic state component values ​​into low-level, specific device parameter values. This process can be represented by the following formula:

[0109]

[0110] In the formula: This generates a vector of device control parameters, which contains the target parameter values ​​for each device that needs to be controlled, for example... Represents the preset inverse mapping function; The input is the optimal target state vector.

[0111] For example, if the value of the "lighting environment" component in the optimal target state vector is 0.8, the inverse mapping function will, according to preset allocation rules and weights, parse it into a series of specific parameter values, such as setting the brightness of the main living room light to 75% and the brightness of the corridor light to 50%. This generates a complete device control parameter vector. Then, the execution display module generates device control commands that conform to the corresponding communication protocol based on the parameters in the vector, and sends them to the corresponding smart devices for execution through the smart home network.

[0112] Regarding the generation and presentation of the integrated display interface, the execution display module not only receives the optimal target state vector It also receives the future state vector generated by the state prediction module. Based on these two vectors, the module generates an information-integrated display interface on the central control screen.

[0113] In one possible implementation, the merged display interface comprises two main parts:

[0114] A state visualization interface based on the optimal target state vector. Rendering is performed. This provides real-time, visual feedback to the user's actions. For example, when a user issues a command to adjust the brightness, the graphical interface representing the indoor environment on the screen will adjust its brightness in real time accordingly, intuitively presenting the effect after the command is executed.

[0115] A predictive visual indication based on the future state vector. The rendering is performed and overlaid or blended into the state visualization interface in a non-intrusive manner. The purpose is to suggest to the user the future natural evolution of the environment. Alternatively, the predictive visual indication can be a subtle animation effect, such as a faint glow, a semi-transparent trend arrow, or a gradient color overlay. For example, if the system predicts that the indoor temperature will naturally rise due to sunlight in the future, a faint, upward-pointing red arrow can be overlaid in the temperature display area to provide the user with forward-looking information.

[0116] By integrating visualizations of the real-time status with predictive indications of future trends, users can not only confirm the results of their current actions but also anticipate the next changes in the environment, thereby gaining a stronger sense of control and certainty.

[0117] Furthermore, the execution display module is also responsible for receiving proactive interaction proposals generated by the state prediction module under specific conditions. When the proposal is received, the module displays it clearly on the central control screen, such as by popping up a dialog box or displaying a notification card. The displayed content explains the potential problems predicted by the system and provides clear interaction options for the user to choose from, thus completing a closed-loop interaction from intelligent prediction to proactive service.

[0118] Please see Figure 2 The present invention also provides a multimodal data interaction display method for a smart home central control screen, the method comprising the following steps:

[0119] S1. Obtain device data of smart home devices, generate a current state vector representing the current environment based on the device data, and record the current state vector to form state history trajectory data;

[0120] The system first comprehensively collects raw data reported by various smart home devices, such as temperature, humidity, brightness, and air quality readings. Then, using a pre-defined aggregation and weighting algorithm, the system integrates and abstracts this scattered underlying data into a unified data structure that macroscopically describes the overall home environment—the current state vector. Simultaneously, the system records each generated current state vector in chronological order, thus forming a historical trajectory of state data reflecting long-term environmental trends.

[0121] S2. Based on multimodal interactive input, generate an intent transformation operator representing the user's intent and an interactive feature vector representing the interactive attributes.

[0122] When users interact with the central control screen through various means such as voice, touch, or gestures, the system deeply analyzes this multimodal interaction input. On one hand, the system extracts the user's core command intent and operation range, such as "turn on the living room lights," and transforms it into a structured instruction that the system can understand to guide state changes—that is, an intent transformation operator. On the other hand, the system also extracts non-semantic physical features from the interaction behavior, such as the user's speaking speed, volume, or touch pressure, and combines them into an interaction feature vector that reflects the user's current interaction state or emotion.

[0123] S3. Based on the current state vector, the intention transformation operator, and the interaction feature vector, the optimal target state vector is generated by solving a constraint optimization model dynamically adjusted by the interaction feature vector.

[0124] The system combines the current state vector representing the current environment with the intent transformation operator representing the user's command to generate a preliminary target state. However, the system does not directly execute this preliminary target; instead, it solves for it using a constrained optimization model. This model comprehensively weighs the user's command satisfaction, the system's energy consumption, the physical security limitations of the devices, and the consistency requirements of the current scenario mode. In particular, the model uses the interaction feature vector generated in the previous step to dynamically adjust the focus of the decision-making; for example, when the user's command is perceived to be urgent, priority is given to ensuring response speed rather than energy saving. Ultimately, the model solves for an optimal target state vector that achieves the best balance among all factors.

[0125] S4. Based on the historical trajectory data of the state, predict and generate future state vectors using a time-series prediction model;

[0126] To achieve forward-looking capabilities, the system utilizes historical state trajectory data accumulated in step S1. A time-series prediction model analyzes the patterns inherent in this historical data, such as periodic changes in light intensity or temperature throughout the day, to predict the state the home environment might naturally evolve into at a future point in time, generating a future state vector. This prediction result forms the basis for proactive services and predictive displays.

[0127] S5. Generate and issue device control commands based on the optimal target state vector, and display the state visualization interface and predictive visual indication corresponding to the optimal target state vector and the future state vector respectively on the central control screen.

[0128] Based on the optimal target state vector calculated in step S3, the system reverse-engineers it into precise control commands for each specific device. For example, it sets the brightness of a lamp to a specific value and issues a command to make the device perform the action. Simultaneously, the central control screen displays integrated information: not only does it visually show the new environmental state after the command execution through a status visualization interface, but it also overlays the environmental change trend represented by the future state vector predicted in step S4 with a non-disturbing visual effect, providing users with both immediate feedback and future predictions.

[0129] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal data interaction display system for a smart home central control screen, characterized in that, The system includes: The state modeling module is used to acquire device data of smart home devices, generate a current state vector representing the current environment based on the device data, and record the current state vector to form state history trajectory data. An interaction parsing module is used to generate an intent transformation operator representing the user's intent and an interaction feature vector representing the interaction attributes based on multimodal interaction input. The interaction parsing module includes: parsing and generating a speech semantic vector and a spatial semantic vector based on the multimodal interaction input; calculating the speech semantic vector and the spatial semantic vector using a fusion generation function to generate the intent transformation operator; and extracting and generating the interaction feature vector based on non-semantic physical features in the multimodal interaction input, wherein the non-semantic physical features include the user's speaking volume, speaking speed, pitch, and the force, speed, and trajectory smoothness of the user's touch or gesture. A decision optimization module is used to combine the current state vector and the intent transformation operator, and solve for and generate an optimal target state vector through a constraint optimization model dynamically adjusted by the interaction feature vector. The decision optimization module includes: applying the intent transformation operator to the current state vector to generate a preliminary target state vector, and using the preliminary target state vector as the optimization objective of the constraint optimization model; solving the constraint optimization model to generate the optimal target state vector; the constraint optimization model includes an objective function configured to minimize a weighted cost function; the weighted cost function is calculated using the preliminary target state vector as input, and includes: a deviation cost term calculated based on the deviation between the optimal target state vector and the preliminary target state vector, and a system operating cost term calculated based on the optimal target state vector; the decision optimization module dynamically adjusts the weight coefficients corresponding to each cost term in the weighted cost function based on the interaction feature vector. The state prediction module is used to predict and generate future state vectors based on the historical trajectory data of the states using a time-series prediction model. The execution display module is used to generate and issue device control commands based on the optimal target state vector, and to integrate and display the state visualization interface and predictive visual indicators corresponding to the optimal target state vector and the future state vector on the central control screen.

2. The multimodal data interaction display system for a smart home central control screen according to claim 1, characterized in that, The state modeling module includes: The device data is subjected to a preset weighted aggregation operation to obtain multiple state components, and the multiple state components are combined into the current state vector; wherein, the state components are used to characterize the macroscopic dimension of the home environment.

3. The multimodal data interaction display system for a smart home central control screen according to claim 1, characterized in that, The constraint optimization model in the decision optimization module further includes constraints, which include: Equipment physical constraints are used to ensure that the equipment parameters corresponding to the optimal target state vector are within a preset safe operating range; Scene mode consistency constraints are used to ensure that the optimal target state vector meets the preset requirements of the current scene mode of the system.

4. The multimodal data interaction display system for a smart home central control screen according to claim 1, characterized in that, The state prediction module includes: The future state vector is compared with a preset user preference model to determine the prediction deviation; and when the prediction deviation meets a preset trigger condition, an active interaction proposal is generated.

5. A multimodal data interaction display system for a smart home central control screen according to claim 4, characterized in that, The execution display module is also used to receive the proactive interaction proposal and display the proactive interaction proposal on the central control screen.

6. The multimodal data interaction display system for a smart home central control screen according to claim 1, characterized in that, The execution display module generates the device control command as follows: The optimal target state vector is used as input to calculate the device control command corresponding to the optimal target state vector through a preset reverse mapping function.

7. A multimodal data interaction display method for a smart home central control screen, applied to the system described in any one of claims 1-6, characterized in that, The method includes the following steps: S1. Obtain device data of smart home devices, generate a current state vector representing the current environment based on the device data, and record the current state vector to form state history trajectory data; S2. Based on multimodal interactive input, generate an intent transformation operator representing the user's intent and an interactive feature vector representing the interactive attributes. S3. Based on the current state vector, the intention transformation operator, and the interaction feature vector, the optimal target state vector is generated by solving a constraint optimization model dynamically adjusted by the interaction feature vector. S4. Based on the historical trajectory data of the state, predict and generate future state vectors using a time-series prediction model; S5. Generate and issue device control commands based on the optimal target state vector, and display the state visualization interface and predictive visual indication corresponding to the optimal target state vector and the future state vector respectively on the central control screen.

Images