Device control method and device based on multi-modal data fusion, and storage medium

Abstract

The application discloses a device control method and device based on multi-modal data fusion and a storage medium, relates to the technical field of smart homes, and comprises the following steps: obtaining multi-modal data obtained by environment perception in a target space; wherein the multi-modal data comprises structured data and unstructured data; fusing the multi-modal data through a preset model to obtain a fusion result; wherein the preset model comprises a feature extraction layer and a feature fusion layer connected with the feature extraction layer; wherein the feature extraction layer is used for extracting data features of the multi-modal data, and the feature fusion layer is used for performing feature fusion on the data features extracted from the multi-modal data; and controlling one or more smart home devices in the target space based on the fusion result.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a device control method and apparatus, and storage medium based on multimodal data fusion. Background Technology

[0002] In related technologies, smart home systems typically rely on a single type of data (such as sensor data or image data) to control smart home devices; and they usually employ simple rules or single-model-based strategies to control these devices. This lack of effective fusion and comprehensive utilization of multimodal data (such as images, sound, and sensor data) results in low system intelligence, an inability to adjust device status in real time, and slow response times. Consequently, the device control strategies for smart home devices are not optimized, leading to a poor user experience.

[0003] There is still no effective solution to the problem that smart home systems struggle to effectively integrate multimodal data, resulting in low levels of system intelligence.

[0004] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0005] This application provides a device control method, apparatus, and storage medium based on multimodal data fusion, to at least solve the problem in related technologies where smart home systems have difficulty effectively integrating multimodal data, resulting in low system intelligence.

[0006] According to one aspect of the embodiments of this application, a device control method based on multimodal data fusion is provided, comprising: acquiring multimodal data obtained from environmental perception within a target space; wherein the multimodal data includes structured data and unstructured data; fusing the multimodal data through a preset model to obtain a fusion result; wherein the preset model includes: a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data; and controlling one or more smart home devices within the target space based on the fusion result.

[0007] In an exemplary embodiment, acquiring multimodal data obtained from environmental perception within a target space includes: acquiring environmental data of the target space collected by sensors, and / or device status data of one or more smart home devices; determining the environmental data and / or the device status data as structured data in the multimodal data; and acquiring image data and / or audio / video data collected by an image acquisition device; determining the image data and / or the audio / video data as unstructured data in the multimodal data.

[0008] In an exemplary embodiment, fusing the multimodal data through a preset model to obtain a fusion result includes: extracting data features of the multimodal data through N neural networks of the feature extraction layer; wherein the N neural networks process the multimodal data in parallel; the N neural networks are respectively used to extract data features of different modal data, and N is a positive integer; inputting the data features into the input interface of the feature fusion layer, and using the output result of the feature fusion layer as the fusion result.

[0009] In an exemplary embodiment, extracting data features from the multimodal data using N neural networks of the feature extraction layer includes: when the number of modalities included in the multimodal data is M, extracting modal data features corresponding to each modality in the multimodal data using M neural networks among the N neural networks; wherein, the M neural networks correspond one-to-one with the multimodal data, and M is a positive integer; and extracting time-series features corresponding to the time-series data using a target neural network among the N neural networks; wherein, the target neural network is a neural network used to extract time-series features; the time-series data is time-series data extracted from the multimodal data; and determining the M modal data features and the time-series features as the data features.

[0010] In an exemplary embodiment, controlling one or more smart home devices within the target space based on the fusion result includes: determining a first state adjustment command for the one or more smart home devices through the fusion result; and controlling the one or more smart home devices through the first state adjustment command when preset conditions are met; wherein the preset conditions include one of the following: a second state adjustment command issued by the target object for the one or more smart home devices does not conflict with the first state adjustment command; or the target object has not issued the second state adjustment command; and the target object is an object that is allowed to control the one or more smart home devices.

[0011] In an exemplary embodiment, determining a first state adjustment instruction for the one or more smart home devices based on the fusion result includes: determining the current spatial state of the target space indicated by the fusion result, wherein the current spatial state includes: the state of the target object; determining the expected spatial state corresponding to the current spatial state; determining the one or more smart home devices that need to be adjusted to adjust the current spatial state to the expected spatial state; and determining the state adjustment instruction for adjusting the one or more smart home devices as the first state adjustment instruction.

[0012] In an exemplary embodiment, controlling one or more smart home devices within the target space based on the fusion result includes: inputting the fusion result into a reinforcement learning network, and outputting a first parameter optimization instruction for the one or more smart home devices through the reinforcement learning network based on the fusion result; and controlling the one or more smart home devices through the first parameter optimization instruction.

[0013] In an exemplary embodiment, after the reinforcement learning network outputs a first parameter optimization instruction for the one or more smart home devices based on the fusion result, the method further includes: inputting a second parameter optimization instruction for the one or more smart home devices into the reinforcement learning network, wherein the second parameter optimization instruction and the first parameter optimization instruction are similar control instructions for the same smart home device, and the first parameter optimization instruction and the second parameter optimization instruction have the same temporal characteristics; and continuing to train the reinforcement learning network using the first parameter optimization instruction and the second parameter optimization instruction.

[0014] According to another aspect of the embodiments of this application, a device control apparatus based on multimodal data fusion is also provided, comprising: an acquisition module, configured to acquire multimodal data obtained from environmental perception within a target space; wherein the multimodal data includes structured data and unstructured data; a fusion module, configured to fuse the multimodal data through a preset model to obtain a fusion result; wherein the preset model includes: a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein the feature extraction layer is configured to extract data features from the multimodal data, and the feature fusion layer is configured to perform feature fusion on the data features extracted from the multimodal data; and a control module, configured to control one or more smart home devices within the target space based on the fusion result.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described device control method based on multimodal data fusion at runtime.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the device control method based on multimodal data fusion through the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described device control method based on multimodal data fusion.

[0018] This application first acquires multimodal data obtained from environmental perception within a target space; wherein the multimodal data includes structured and unstructured data; secondly, the multimodal data is fused using a preset model to obtain a fusion result; wherein the preset model includes a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data; finally, after the data feature fusion of the multimodal data is completed, one or more smart home devices within the target space can be controlled based on the fusion result. Therefore, by adopting the above technical solution, the problem of low system intelligence caused by the difficulty in effectively fusing multimodal data in related technologies is solved; furthermore, the intelligence level of the smart home system is improved by effectively fusing multimodal data, thereby enhancing the user experience. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the hardware environment of an optional device control method based on multimodal data fusion according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of an optional device control method based on multimodal data fusion according to an embodiment of this application;

[0023] Figure 3This is a smart home system architecture diagram of an optional device control method based on multimodal data fusion according to an embodiment of this application;

[0024] Figure 4 This is a structural block diagram of an optional device control apparatus based on multimodal data fusion according to an embodiment of this application;

[0025] Figure 5 This is another structural block diagram of an optional device control apparatus based on multimodal data fusion according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to one aspect of the embodiments of this application, a device control method based on multimodal data fusion is provided. This device control method based on multimodal data fusion is widely used in whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligencehouse ecosystems. Optionally, in this embodiment, the above-mentioned device control method based on multimodal data fusion can be applied to, for example... Figure 1 The hardware environment shown consists of multiple terminal devices 102 and a server 104. For example... Figure 1As shown, server 104 is connected to multiple terminal devices 102 via a network and can be used to provide services (such as application services) to terminals or clients installed on terminals. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0030] This embodiment provides a device control method based on multimodal data fusion, including but not limited to applications in smart home systems. Figure 2 This is a flowchart of a device control method based on multimodal data fusion according to an embodiment of this application. The process includes the following steps:

[0031] Step S202: Acquire multimodal data obtained from environmental perception within the target space; wherein, the multimodal data includes: structured data and unstructured data;

[0032] The structured data is numerical data related to the target space, and the unstructured data is non-numerical data related to the target space.

[0033] Step S204: The multimodal data is fused using a preset model to obtain a fusion result; wherein the preset model includes: a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data;

[0034] Step S206: Based on the fusion result, control one or more smart home devices in the target space.

[0035] Through the above steps, firstly, multimodal data of environmental perception within the target space is acquired; wherein, the multimodal data includes structured data and unstructured data; secondly, the multimodal data is fused through a preset model to obtain a fusion result; wherein, the preset model includes a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein, the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data; finally, after the data feature fusion of the multimodal data is completed, one or more smart home devices within the target space can be controlled based on the fusion result. Therefore, by adopting the above technical solution, the problem of low system intelligence caused by the difficulty in effectively fusing multimodal data in related technologies is solved; thereby, the intelligence level of the smart home system is improved by effectively fusing multimodal data, thus enhancing the user experience.

[0036] In an exemplary embodiment, acquiring multimodal data of a target space containing multiple smart home devices includes: acquiring multimodal data obtained from environmental perception within the target space, including: acquiring environmental data of the target space collected by sensors, and / or device status data of the one or more smart home devices; determining the environmental data and / or the device status data as structured data in the multimodal data; and acquiring image data and / or audio / video data collected by an image acquisition device; determining the image data and / or the audio / video data as unstructured data in the multimodal data.

[0037] Optionally, sensors can be directly deployed on various smart home devices or directly communicate with the smart home system. Examples of sensors include temperature and humidity sensors, and light sensors. Environmental data collected by sensors includes, but is not limited to, temperature, humidity, and light data of the target environment. Environmental data and device status data are data that can be directly represented in numerical form, and are therefore considered structured data within the multimodal data. Non-numerical data, such as image data, video data, and audio data collected from various smart home devices like cameras and microphones, are considered unstructured data within the multimodal data. This application's embodiments, by acquiring and processing multimodal data, enable the smart home system to integrate visual, auditory, and environmental information, providing a richer information set than single-modal data processing.

[0038] In an exemplary embodiment, fusing the multimodal data through a preset model to obtain a fusion result includes: extracting data features of the multimodal data through N neural networks of the feature extraction layer; wherein the N neural networks process the multimodal data in parallel; the N neural networks are respectively used to extract data features of different modal data, and N is a positive integer; inputting the data features into the input interface of the feature fusion layer, and using the output result of the feature fusion layer as the fusion result.

[0039] The feature extraction layer can include N parallel neural networks. The input interfaces of the N neural networks are connected to the input interface of the feature extraction layer, and the output interfaces of the N neural networks are connected to the output interface of the feature extraction layer. For example, one neural network might be specifically designed for processing image data, another for processing audio data, and yet another for processing environmental sensor data (such as temperature and humidity). Optionally, the feature fusion layer can use a Transformer model. The feature fusion layer integrates the features extracted by different neural networks to form a unified, comprehensive feature representation. This is typically achieved through concatenation, splicing, weighted summation, or more complex fusion methods (such as multimodal attention mechanisms). The fused features are more comprehensive, containing complementary information from the multimodal data, and can provide richer input for subsequent decision-making and control.

[0040] Through the steps described above, smart home systems can effectively process and fuse multimodal data from different sensors and devices to form a comprehensive description of the environment and user behavior. These fused features can then be fed into reinforcement learning models or other decision-making models to optimize device control strategies, thereby achieving goals such as improving user experience, saving energy, or enhancing security monitoring.

[0041] Furthermore, extracting data features from the multimodal data using N neural networks in the feature extraction layer includes: when the number of modalities included in the multimodal data is M, extracting modal data features corresponding to each modality in the multimodal data using M neural networks among the N neural networks; wherein, the M neural networks correspond one-to-one with the multimodal data, and M is a positive integer; and extracting time-series features corresponding to the time-series data using a target neural network among the N neural networks; wherein, the target neural network is a neural network used to extract time-series features; the time-series data is time-series data extracted from the multimodal data; and determining the M modal data features and the time-series features as the data features.

[0042] It is understandable that the specific feature extraction process of N neural networks for multimodal data includes: extracting features corresponding to each modality in the multimodal data through M neural networks, and extracting the temporal features of the corresponding time series data through a target neural network, such as a recurrent neural network. It is also understandable that multimodal data carries time series data.

[0043] In some optional embodiments, controlling one or more smart home devices within the target space based on the fusion result includes: determining a first state adjustment command for the one or more smart home devices through the fusion result; and controlling the one or more smart home devices through the first state adjustment command when preset conditions are met; wherein the preset conditions include one of the following: a second state adjustment command issued by the target object for the one or more smart home devices does not conflict with the first state adjustment command; or the target object has not issued the second state adjustment command; and the target object is an object that is allowed to control the one or more smart home devices.

[0044] The fusion result refers to the comprehensive information obtained by the system through the fusion of multimodal data, which includes a deep understanding of the target space environment, user behavior, and device status. Based on the fusion result, the system can then generate initial state adjustment commands for various smart home devices, such as adjusting the air conditioner temperature or changing the brightness of lights, to optimize environmental conditions or improve the user experience.

[0045] Optionally, determining a first state adjustment instruction for the one or more smart home devices based on the fusion result includes: determining the current spatial state of the target space indicated by the fusion result, wherein the current spatial state includes: the state of the target object; determining the expected spatial state corresponding to the current spatial state; determining the one or more smart home devices that need to be adjusted to adjust the current spatial state to the expected spatial state; and determining the state adjustment instruction for adjusting the one or more smart home devices as the first state adjustment instruction.

[0046] Optionally, the expected spatial state is predicted by a machine learning model based on historical spatial states. This machine learning model is trained using historical spatial states and the corresponding expected spatial states. Alternatively, the expected spatial state can be pre-defined based on time conditions, which include at least one of the following: season, time of day, or day of the week.

[0047] Specifically, by analyzing the fusion results, the current spatial state of the target space can be determined, including environmental parameters (such as temperature and humidity), device status (such as whether the air conditioner is on and the position of the curtains), and user behavior (such as being in the room and watching TV). For example, it may be identified that the current temperature is 26°C, the user is resting in the room, and the curtains are half-open.

[0048] Optionally, for the fusion result: The fusion result may be a multi-dimensional vector, where each dimension corresponds to a feature or information type. For example, for a smart home system, the fusion result vector may contain the following dimensions: temperature features, humidity features, light features, sound features, image features (such as the user's location in the room, ongoing activities), device status features (such as the current status of air conditioners, lights, and curtains), etc. Each feature is extracted from the raw data and converted into a numerical representation by the corresponding model (such as CNN, RNN, Transformer). These feature vectors are integrated into a single vector in the fusion layer using various methods (such as weighted summation, concatenation, attention mechanisms) to form the fusion result. Optionally, the fusion result can be input into a machine learning or reinforcement learning model, which determines the current spatial state based on the environmental state and user behavior information contained in the fusion result.

[0049] Then, machine learning models, such as neural network models, trained using historical data (including historical spatial states), are used to predict the expected spatial state of the target space. The expected spatial state is the ideal state that the system anticipates achieving based on user preferences, environmental changes, and equipment capabilities. For example, if historical data shows that users tend to set the temperature to 24°C and completely close the curtains when resting, the system will predict these as the expected spatial state.

[0050] Then, by comparing the current spatial state with the expected spatial state, it determines which devices need to be adjusted to achieve the expected state. For example, if the current temperature is 26℃ and the expected temperature is 24℃, it decides to adjust the air conditioner to lower the temperature; if the curtains are currently half-open but the expected state is for the curtains to be completely closed, it decides to adjust the curtains. After determining the devices that need adjustment, specific first-state adjustment instructions are generated to control the devices to adjust their states. These instructions can be specific device operations (such as "adjust the air conditioner temperature to 24℃" or "completely close the curtains").

[0051] Furthermore, before executing the first state adjustment command, the smart home system will check whether preset conditions are met. If they are met, the system will execute the first state adjustment command to adjust the state of the smart home devices. If they are not met, for example, if the user command conflicts with the system command, the system should prioritize executing the user command.

[0052] This embodiment demonstrates how a smart home system can strike a balance between autonomous control and user participation, ensuring that system-generated control commands consider both the overall state of the environment and devices, as well as the user's direct needs and preferences. This approach not only enhances the intelligence and autonomy of system decision-making but also strengthens user trust and satisfaction.

[0053] In some optional embodiments, controlling one or more smart home devices within the target space based on the fusion result includes: inputting the fusion result into a reinforcement learning network, and outputting a first parameter optimization instruction for the one or more smart home devices through the reinforcement learning network based on the fusion result; and controlling the one or more smart home devices through the first parameter optimization instruction.

[0054] This application utilizes the results of multimodal data fusion to optimize the parameters of smart home devices through a reinforcement learning network, thereby achieving smarter and more personalized home environment management. Specifically, the fusion results are input into a pre-trained reinforcement learning network. Reinforcement learning is a machine learning technique that allows a model (agent) to learn how to take actions to maximize a certain reward signal through interaction with the environment, which could be goals such as improving user comfort, energy saving, or security monitoring. Based on the fusion results, the reinforcement learning network outputs first parameter optimization instructions for the smart home devices through its internal decision-making mechanism. These instructions can be specific device control parameters, such as air conditioning temperature, light brightness, and the degree of curtain opening. By simulating interaction with the environment, the reinforcement learning network predicts how these parameter adjustments will affect the environmental state and user satisfaction, thereby outputting the optimal control strategy. The (smart home) system sends the first parameter optimization instructions to the corresponding smart home devices to adjust their operating parameters. For example, if the fusion results indicate that the user is watching a movie in dim lighting, the reinforcement learning network might output instructions to reduce light brightness and increase TV volume to create a more suitable viewing environment.

[0055] The key to implementing this solution lies in the fact that reinforcement learning networks can learn and adapt to user behavior patterns and preferences, as well as environmental changes. Over time, it will continuously optimize its decision-making strategies, thereby more accurately predicting user needs and automatically adjusting device parameters to provide a more comfortable and energy-efficient smart home environment without requiring frequent user interaction. This reinforcement learning-based multimodal data fusion control method can not only handle complex multimodal information but also improve the intelligence and flexibility of device control through autonomous learning.

[0056] Furthermore, after the reinforcement learning network outputs a first parameter optimization instruction for the one or more smart home devices based on the fusion result, the method further includes: inputting a second parameter optimization instruction for the one or more smart home devices into the reinforcement learning network, wherein the second parameter optimization instruction and the first parameter optimization instruction are similar control instructions for the same smart home device, and the first parameter optimization instruction and the second parameter optimization instruction have the same temporal characteristics; and continuing to train the reinforcement learning network using the first parameter optimization instruction and the second parameter optimization instruction.

[0057] Optionally, the reinforcement learning network deployment steps in this application embodiment are as follows:

[0058] Define the environment: The environment is the physical or virtual system in which the intelligent agent interacts. In the context of smart homes, the environment typically includes all elements that can be influenced and controlled by devices. For example: State Space: Defines the environmental states that the intelligent agent can observe, such as indoor temperature, humidity, light intensity, user location, activity patterns, and current device energy consumption. The state space can be continuous or discrete, depending on the specific application requirements. Reward Signal: Sets the reward mechanism to quantify the quality of the agent's behavior. Rewards can be immediate (e.g., increased user satisfaction) or long-term (e.g., energy savings).

[0059] Design Action Space: The action space defines the actions an agent can perform. In a smart home device control scenario, actions might include: adjusting the air conditioner's temperature setting; controlling the brightness or color of lights; and turning specific devices on or off. Depending on the device's characteristics and application scenario, the action space can be continuous (e.g., any value within the temperature adjustment range) or discrete (e.g., turning devices on / off).

[0060] Agent Model Selection: The agent model is the learning component in the RL framework, responsible for selecting actions based on the current state and historical experience. In smart home control, model-based RL (such as planning-based methods) or model-free RL (such as Qling, DQN, PPO, etc.) can be used.

[0061] Designing policy learning algorithms: Policy learning algorithms determine how an agent updates its decision-making policy. Common algorithms include: Qlearning: Applicable to discrete action spaces, it updates the agent's Q-value to learn to choose the best action in a given state. DQN (Deep Q-Network): Building upon Qlearning, it uses a deep neural network to estimate the Q-value, suitable for more complex state and action spaces. PPO (Proximal Policy Optimization): A policy gradient method suitable for continuous action spaces, it maximizes cumulative reward by optimizing the policy function. A3C (Asynchronous Advantage Actor-Critic): A parallel and asynchronous policy gradient method suitable for large-scale and complex environments.

[0062] Exploration vs. Exploitation: During the learning process, an agent needs to find a balance between exploring new strategies and utilizing known best strategies. A common approach is to use an ε-greedy strategy, which randomly selects actions with a certain probability to explore the environment, and spends the rest of the time choosing the action that the current strategy considers best.

[0063] Training the agent: The agent is trained using reinforcement learning algorithms. It interacts with the environment multiple times, trying different action strategies and adjusting its strategy based on reward signals. Training typically involves numerous iterations until the policy converges or reaches a preset performance standard.

[0064] Evaluation and Optimization: During training, the agent's performance is evaluated periodically to ensure it optimizes towards the intended goal. This may require a validation environment or dataset to unbiasedly test the agent's performance. Based on the evaluation results, the agent model, reward function, exploration strategy, or other parameters may need to be adjusted.

[0065] Deploying the agent: After training, the agent is deployed in a real-world environment, its performance is monitored, and necessary fine-tuning is performed. In a smart home system, this means the agent will dynamically adjust the device's control strategy based on real-time data and user feedback (e.g., second-parameter optimization instructions).

[0066] A key characteristic of reinforcement learning is its ability to learn continuously. Device control policies need to be adjusted based on environmental changes, updated user behavior patterns, or the addition of new devices. The agent can continue to learn and optimize its policies after deployment. Before actual deployment, it is necessary to ensure that the agent's decisions do not lead to unstable device operation or threaten user safety. This may involve additional testing and verification steps for the agent's policies.

[0067] Through the steps described above, reinforcement learning can be applied to smart home systems to intelligently optimize device control strategies, improve system efficiency and user experience, and reduce energy consumption. Deploying reinforcement learning is an iterative and optimization process that requires continuous adjustment and balance between theoretical research, experimental design, and practical application.

[0068] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above-described device control method based on multimodal data fusion, the process is described below in conjunction with embodiments, but this is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0069] This application proposes a smart home system based on multimodal data fusion, which can be used to implement the device control method based on multimodal data fusion in the above embodiments. For example... Figure 3 As shown, the architecture of this smart home system includes a data collection and storage module, a data preprocessing and fusion module, a model training and prediction module, a real-time control and optimization module, and a user interaction module.

[0070] Specifically, data collection and storage involves gathering multimodal data from environmental sensors, device status, user behavior (i.e., structured data), and images, videos, and audio (i.e., unstructured data). All collected multimodal data is stored in a unified data storage system for subsequent processing.

[0071] The data preprocessing and fusion process includes: cleaning the collected multimodal data to remove noise and outliers; normalizing the multimodal data to make it suitable for model input; temporal alignment of multimodal data from different sources to ensure temporal consistency; and using a multimodal deep learning network to fuse the multimodal data from different sources to generate a unified feature representation.

[0072] Among them, model training and prediction:

[0073] Image data is processed using a Convolutional Neural Network (CNN) to extract image features, and these features are then input into a multimodal data fusion module. Time-series data is processed using a Recurrent Neural Network (RNN) to extract temporal features, and these features are then input into the multimodal data fusion module.

[0074] The Transformer model is used to perform multimodal data fusion and complex time series prediction, generating prediction results (equivalent to the fusion results in the above embodiments), which are then input into the predictive adjustment module.

[0075] Reinforcement learning (RL) is used to optimize the device control strategy, generating an optimized strategy which is then input into the parameter optimization module.

[0076] Among them, real-time control and optimization:

[0077] Adjust the equipment status in advance based on the forecast results.

[0078] Based on the optimization strategy, the equipment parameters are automatically adjusted to achieve optimal control.

[0079] Among them, user (i.e., target object) interaction:

[0080] Users can activate smart home functions via voice or mobile application.

[0081] Users input their needs via voice or mobile application, and the system controls the devices and optimizes parameters accordingly.

[0082] The present application will be further explained and illustrated by the following two embodiments.

[0083] Example 1: Intelligent temperature control system.

[0084] Data collection: Indoor environmental data is collected through temperature, humidity and light sensors; user behavior and voice commands are monitored through cameras and microphones.

[0085] Data preprocessing: The collected data is cleaned, normalized, and time-series aligned, and then fused using a multimodal Transformer.

[0086] Model training and prediction: CNN is used to extract image features, RNN is used to extract temporal features, Transformer is used for multimodal data fusion and temporal prediction, and reinforcement learning is used to optimize device control strategies.

[0087] Real-time control: Based on forecast results and optimization strategies, adjust the air conditioning temperature in advance to provide a comfortable home environment.

[0088] User interaction: Users input their needs through a voice assistant, the system adjusts the air conditioner temperature accordingly, and provides feedback on the adjustment results through a mobile application.

[0089] Example 2: Intelligent lighting system.

[0090] Data collection: Ambient lighting and user behavior data are collected through light sensors, cameras, and microphones.

[0091] Data preprocessing: The collected data is cleaned, normalized, and time-series aligned, and then fused using a multimodal Transformer.

[0092] Model training and prediction: CNN is used to extract image features, RNN is used to extract temporal features, Transformer is used for multimodal data fusion and temporal prediction, and reinforcement learning is used to optimize device control strategies.

[0093] Real-time control: Automatically adjusts lighting brightness based on prediction results and optimization strategies to provide a suitable lighting environment.

[0094] User interaction: Users input their needs through a voice assistant, the system adjusts the lighting brightness accordingly, and provides feedback on the adjustment results through a mobile application.

[0095] In summary, this application embodiment utilizes multimodal data fusion technology to comprehensively utilize multiple data sources such as images, sound, and sensor data, thereby enhancing the system's intelligence. Through optimized device control strategies and real-time adjustment mechanisms, the system can respond to user needs more quickly and provide a better user experience. Compared to existing technologies, this application embodiment demonstrates significant improvements in data processing, device control, and user interaction.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0097] This embodiment also provides a device control apparatus based on multimodal data fusion, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0098] Figure 4 This is a structural block diagram of an optional device control apparatus based on multimodal data fusion according to an embodiment of this application. The apparatus includes:

[0099] The acquisition module 42 is used to acquire multimodal data obtained from environmental perception within the target space; wherein, the multimodal data includes: structured data and unstructured data;

[0100] The fusion module 44 is used to fuse the multimodal data through a preset model to obtain a fusion result; wherein the preset model includes: a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data;

[0101] Control module 46 is used to control one or more smart home devices in the target space based on the fusion result.

[0102] The above-described device first acquires multimodal data obtained from environmental perception within a target space. This multimodal data includes structured and unstructured data. Second, the multimodal data is fused using a preset model to obtain a fusion result. This preset model includes a feature extraction layer and a feature fusion layer connected to the feature extraction layer. The feature extraction layer extracts data features from the multimodal data, and the feature fusion layer performs feature fusion on the extracted data features. Finally, after the multimodal data feature fusion is complete, one or more smart home devices within the target space can be controlled based on the fusion result. Therefore, this technical solution solves the problem of low system intelligence caused by the difficulty in effectively fusing multimodal data in related technologies. Furthermore, by effectively fusing multimodal data, the intelligence level of the smart home system is improved, thereby enhancing the user experience.

[0103] In an exemplary embodiment, the acquisition module 42 is further configured to: acquire environmental data of the target space collected by sensors, and / or device status data of the one or more smart home devices; determine the environmental data and / or the device status data as structured data in the multimodal data; and acquire image data and / or audio-visual data collected by the image acquisition device; determine the image data and / or the audio-visual data as unstructured data in the multimodal data.

[0104] In an exemplary embodiment, the fusion module 44 is further configured to: extract data features of the multimodal data through N neural networks of the feature extraction layer; wherein the N neural networks process the multimodal data in parallel; the N neural networks are respectively used to extract data features of different modal data, and N is a positive integer; input the data features into the input interface of the feature fusion layer, and use the output result of the feature fusion layer as the fusion result.

[0105] In an exemplary embodiment, the fusion module 44 is further configured to: extract modal data features corresponding to each modality in the multimodal data by means of M neural networks among the N neural networks when the number of modalities included in the multimodal data is M; wherein, the M neural networks correspond one-to-one with the multimodal data, and M is a positive integer; and extract time-series features corresponding to the time-series data by means of a target neural network among the N neural networks; wherein, the target neural network is a neural network used to extract time-series features; the time-series data is time-series data extracted from the multimodal data; and determine the M modal data features and the time-series features as the data features.

[0106] In an exemplary embodiment, the control module 46 is further configured to: determine a first state adjustment instruction for the one or more smart home devices based on the fusion result; and control the one or more smart home devices through the first state adjustment instruction if preset conditions are met; wherein the preset conditions include one of the following: a second state adjustment instruction for the one or more smart home devices issued by the target object does not conflict with the first state adjustment instruction, or the target object has not issued the second state adjustment instruction; and the target object is an object that is allowed to control the one or more smart home devices.

[0107] In an exemplary embodiment, the control module 46 is further configured to: determine the current spatial state of the target space indicated by the fusion result, wherein the current spatial state includes: the state of the target object; determine the expected spatial state corresponding to the current spatial state, wherein the expected spatial state is predicted by a machine learning model based on historical spatial states; determine the one or more smart home devices that need to be adjusted to adjust the current spatial state to the expected spatial state; and determine the state adjustment instruction for adjusting the one or more smart home devices as the first state adjustment instruction.

[0108] In an exemplary embodiment, the control module 46 is further configured to: input the fusion result into a reinforcement learning network, and output a first parameter optimization instruction for the one or more smart home devices based on the fusion result through the reinforcement learning network; and control the one or more smart home devices through the first parameter optimization instruction.

[0109] In one exemplary embodiment, such as Figure 5 As shown, the device further includes: an optimization module 48, configured to: after outputting a first parameter optimization instruction for the one or more smart home devices through the reinforcement learning network based on the fusion result; input a second parameter optimization instruction for the target object on the one or more smart home devices into the reinforcement learning network, wherein the second parameter optimization instruction and the first parameter optimization instruction are similar control instructions for the same smart home device, and the first parameter optimization instruction and the second parameter optimization instruction have the same temporal characteristics; and continue to train the reinforcement learning network through the first parameter optimization instruction and the second parameter optimization instruction.

[0110] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0111] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0112] S1, acquire multimodal data obtained from environmental perception within the target space; wherein, the multimodal data includes: structured data and unstructured data;

[0113] S2, the multimodal data is fused using a preset model to obtain a fusion result; wherein, the preset model includes: a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein, the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data;

[0114] S3, based on the fusion result, control one or more smart home devices in the target space.

[0115] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0116] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0117] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0118] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0119] S1, acquire multimodal data obtained from environmental perception within the target space; wherein, the multimodal data includes: structured data and unstructured data;

[0120] S2, the multimodal data is fused using a preset model to obtain a fusion result; wherein, the preset model includes: a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein, the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data;

[0121] S3, based on the fusion result, control one or more smart home devices in the target space.

[0122] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0123] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0124] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0125] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0126] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0127] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0128] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A device control method based on multimodal data fusion, characterized in that, include: Acquire multimodal data obtained from environmental perception within the target space; wherein, the multimodal data includes: structured data and unstructured data; The multimodal data is fused using a preset model to obtain a fusion result; wherein the preset model includes: a feature extraction layer and a feature fusion layer connected to the feature extraction layer; wherein the feature extraction layer is used to extract data features from the multimodal data, and the feature fusion layer is used to perform feature fusion on the data features extracted from the multimodal data; Based on the fusion results, control one or more smart home devices within the target space.

2. The device control method based on multimodal data fusion according to claim 1, characterized in that, Acquire multimodal data of environmental perception within the target space, including: Acquire environmental data of the target space collected by sensors, and / or device status data of one or more smart home devices; determine the environmental data and / or the device status data as structured data in the multimodal data; and Acquire image data and / or audio / video data collected by the image acquisition device; identify the image data and / or the audio / video data as unstructured data in the multimodal data.

3. The device control method based on multimodal data fusion according to claim 1, characterized in that, The multimodal data is fused using a preset model to obtain a fusion result, including: The feature extraction layer extracts data features from the multimodal data using N neural networks; wherein the N neural networks process the multimodal data in parallel; and the N neural networks are used to extract data features from different modalities, where N is a positive integer. The data features are input into the input interface of the feature fusion layer, and the output of the feature fusion layer is used as the fusion result.

4. The device control method based on multimodal data fusion according to claim 3, characterized in that, The data features of the multimodal data are extracted through N neural networks in the feature extraction layer, including: When the number of modalities included in the multimodal data is M, the modal data features corresponding to each modal data in the multimodal data are extracted by M neural networks out of the N neural networks; wherein, the M neural networks correspond one-to-one with the multimodal data, and M is a positive integer; and The time-series features corresponding to the time-series data are extracted through the target neural network among the N neural networks; wherein, the target neural network is a neural network used to extract time-series features; and the time-series data is time-series data extracted from the multimodal data. The M modal data features and the temporal features are determined as the data features.

5. The device control method based on multimodal data fusion according to claim 1, characterized in that, Based on the fusion result, control one or more smart home devices within the target space, including: The fusion result is used to determine a first state adjustment instruction for the one or more smart home devices; Under preset conditions, the first state adjustment command controls one or more smart home devices; wherein, the preset conditions include one of the following: the second state adjustment command issued by the target object to the one or more smart home devices does not conflict with the first state adjustment command, or the target object does not issue the second state adjustment command; the target object is an object that is allowed to control the one or more smart home devices.

6. The device control method based on multimodal data fusion according to claim 5, characterized in that, The first state adjustment instruction for the one or more smart home devices is determined based on the fusion result, including: Determine the current spatial state of the target space indicated by the fusion result; wherein the current spatial state includes: the state of the target object; Determine the expected spatial state corresponding to the current spatial state; Determine the one or more smart home devices that need to be adjusted to adjust the current spatial state to the expected spatial state; The instruction to adjust the status of one or more smart home devices is determined as the first status adjustment instruction.

7. The device control method based on multimodal data fusion according to claim 1, characterized in that, Based on the fusion result, control one or more smart home devices within the target space, including: The fusion result is input into a reinforcement learning network, and the reinforcement learning network outputs a first parameter optimization instruction for the one or more smart home devices based on the fusion result. The one or more smart home devices are controlled by the first parameter optimization instruction.

8. The device control method based on multimodal data fusion according to claim 7, characterized in that, After the reinforcement learning network outputs a first parameter optimization instruction for the one or more smart home devices based on the fusion result, the method further includes: The second parameter optimization instruction for the target object on one or more smart home devices is input into the reinforcement learning network, wherein the second parameter optimization instruction and the first parameter optimization instruction are similar control instructions for the same smart home device, and the first parameter optimization instruction and the second parameter optimization instruction have the same temporal characteristics. The reinforcement learning network continues to be trained using the first parameter optimization instruction and the second parameter optimization instruction.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 8.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 8 through the computer program.

Images