Multi-modal data active recommendation method based on cross-modal dynamic fusion and equipment resource awareness

Abstract

The invention provides a multi-modal data active recommendation method based on cross-modal dynamic fusion and equipment resource perception. The method is applied to the technical field of data processing. The method comprises the following steps: acquiring multi-modal data through a smart home device end; performing modal feature extraction on the multi-modal data to generate a voice feature matrix, an environment probability matrix and an equipment state vector matrix; adopting a cross-modal dynamic fusion algorithm, combining time dimension and space dimension weights, and carrying out adaptive weighted fusion on the voice feature matrix, the environment probability matrix and the equipment state vector matrix; the CPU load and the memory occupancy rate of the central control equipment are detected in real time through the equipment resource monitoring module, and a data calculation strategy is dynamically adjusted; and based on the fused multi-modal feature matrix, generating an active recommendation instruction through user portrait modeling and knowledge graph association, and driving the smart home device to execute adaptive adjustment. In this way, the recommendation efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method for actively recommending multimodal data based on cross-modal dynamic fusion and device resource perception. Background Art

[0002] Active recommendations based on multimodal data through dynamic cross-modal fusion and device resource awareness primarily generate personalized recommendations by dynamically fusing multimodal data of different types and sources, combined with device resource awareness information. Cross-modal fusion refers to the integration of data from different modalities (such as images, text, and audio) to more comprehensively understand user needs and context. This fusion helps the system acquire information from multiple perspectives, avoiding the limitations of traditional single-modal recommendation systems and improving the accuracy and reliability of recommendations. Dynamic fusion means that the system can flexibly adjust the data fusion strategy and approach as time, environment, and user needs change. For example, at certain times, image information may be more important than text information, while at other times, user behavior data may be more critical. By dynamically adjusting the cross-modal fusion approach, the recommendation system can consistently and stably provide optimal recommendations. Device resource awareness refers to the process by which the system optimizes based on the performance and resource availability of different devices (such as processing power, storage capacity, and network bandwidth). With the rapid development of IoT technology, modern devices often have multiple sensing modules, such as formaldehyde, carbon dioxide, humidity, infrared intrusion, and wind and rain sensors. Combining these device resources with the recommendation system can provide the system with more information and enable resource sharing and collaboration among multiple devices.

[0003] Existing smart home recommendation systems suffer from widespread response delays caused by inefficient multimodal data processing and rigid resource allocation. These issues manifest themselves in the following ways: Inefficient multimodal data collaboration: Traditional systems often employ single-modal analysis or simple weighted fusion strategies, failing to fully consider the spatiotemporal dynamics of voice commands, environmental parameters, and device states in smart home scenarios. This results in a single dimension for cross-modal feature fusion and high computational redundancy. Rigid computing resource allocation mechanisms: Fixed-cycle computing strategies are often employed, lacking dynamic awareness of the real-time load of central control devices. Experimental data shows that when the CPU load exceeds 70%, the recommendation delay of traditional systems increases exponentially, especially in multi-device concurrent scenarios, where fluctuations in memory usage lead to over 30% wasted computing resources. Delayed user profile updates: Mainstream knowledge graph construction methods rely on offline batch processing, making it difficult to promptly reflect dynamic changes in user behavior patterns. Summary of the Invention

[0004] The present disclosure provides a method for proactively recommending multimodal data based on dynamic cross-modal fusion and device resource awareness. The method includes:

[0005] S1, obtains multimodal data through smart home devices, including voice data, environmental sensor data, and status data of user-actively controlled devices;

[0006] S2, performing modal feature extraction on the multimodal data to generate a speech feature matrix, an environment probability matrix, and a device state vector matrix;

[0007] S3, using a cross-modal dynamic fusion algorithm, combining the time dimension and space dimension weights, to perform adaptive weighted fusion on the speech feature matrix, the environment probability matrix, and the device state vector matrix;

[0008] S4: Use the device resource monitoring module to monitor the CPU load and memory usage of the central control device in real time and dynamically adjust the data calculation strategy;

[0009] S5, based on the fused multimodal feature matrix, generates active recommendation instructions through user portrait modeling and knowledge graph association, driving smart home devices to perform adaptive adjustments.

[0010] Furthermore, the method for generating the speech feature matrix in S2 includes:

[0011] Converting the user's voice into text data through voice recognition technology to obtain first business data;

[0012] Performing vectorization processing on the first business data using a word embedding model to generate a word vector sequence;

[0013] A Relu activation function is used to perform nonlinear transformation on the word vector sequence to extract speech semantic features;

[0014] The time dimension is divided into unit time periods, the mean of the word vector in each time period is calculated, and a time-semantic dual-dimensional matrix is ​​constructed as the speech feature matrix.

[0015] Furthermore, the method for generating the environmental probability matrix in S2 includes:

[0016] Acquire indoor temperature and formaldehyde concentration data from security equipment or environmental sensors to obtain second business data;

[0017] Normalizing the second service data and extracting statistical features;

[0018] The Softmax function is used to calculate the reporting probability of each environmental indicator within the preset time period, and an environmental probability matrix with probability values ​​as elements is generated;

[0019] When the environmental comfort level is standard, the matrix value with a higher probability is used as the recommended target; when the security, formaldehyde or non-comfort environment value is abnormal, the recommended value of the recommendation algorithm is triggered.

[0020] Furthermore, the method for generating the device state vector matrix in S2 includes:

[0021] Obtaining the working status data of the device actively controlled by the user to obtain the third service data;

[0022] Perform one-hot encoding and standardization on the device switch status and parameter setting values;

[0023] Convert the processed data into a vector of parameter quantity dimension and construct the device state vector matrix;

[0024] The matrix is ​​updated through a sliding window mechanism to retain the device state change trajectory of the last N time units.

[0025] Furthermore, S3's cross-modal dynamic fusion algorithm includes:

[0026] Calculate the weight coefficients of voice, environment and device modality based on the attention mechanism:

[0027] attention_weights=softmax(MLP(concat(voice_emb,env_emb,device_emb)))

[0028] Perform weighted summation of multimodal features by weight:

[0029] fused_feature=Σ(attention_weights[i]*modality_emb[i])

[0030] Combined with the spatial dimension weight matrix, the fusion features are spatially correlated, where the weight of devices in the same room is 1, the weight of devices on the same floor is 0.25, and the rest are 0;

[0031] A dynamic time warping algorithm based on DTW is used to align the real-time voice data with the sensor data.

[0032] Furthermore, S4's device resource monitoring strategy includes:

[0033] When the CPU usage of the central control device is detected to be ≥90%, the calculation task will be postponed for 5 minutes; if the resource conditions are still not met after three consecutive detection cycles, the calculation will be forced to start and the monitoring cycle will be reset;

[0034] When receiving a JVM low-memory warning, the main algorithm sends a pause instruction for 10 minutes to trigger the garbage collection mechanism;

[0035] After resources are restored, delayed high-priority recommended tasks are executed first.

[0036] Furthermore, S5's user portrait modeling method includes:

[0037] Constructing voice interaction history features: Using the Transformer encoder to extract semantic vectors of the user's high-frequency commands;

[0038] Constructing environmental behavior pattern features: using a temporal convolutional network (TCN) to analyze the temporal changes in temperature and light;

[0039] Build device usage habit features: Model device linkage relationships based on graph neural networks and generate a device operation frequency matrix;

[0040] The three-dimensional features are input into the multimodal fusion layer, and user instructions, device operation records and environmental status are associated through the knowledge graph.

[0041] Furthermore, knowledge graph associations include:

[0042] Map the user voice command "dim the lights" to a node, and associate it with the lighting device node and the light sensor node;

[0043] When a "a bit cold" command is detected, a temperature increase recommendation command is generated based on the temperature and humidity sensor data and the historical set temperature of the air conditioner;

[0044] Identify cross-device linkage rules through the graph inference engine.

[0045] Furthermore, the smart home device performs adaptive adjustment including:

[0046] Choose a 2D / 3D visualization solution based on the device's graphics rendering capabilities;

[0047] When the recommended instruction is triggered, a floating window containing action suggestions is generated;

[0048] The UI layout is optimized based on user feedback data, and high-frequency operation instructions are prioritized for top display.

[0049] Furthermore: when the "sleep mode" keyword is detected in the voice feature matrix, the lights are turned off and the air conditioner speed is lowered;

[0050] When the probability of formaldehyde concentration exceeding the standard in the environmental probability matrix is ​​greater than 85%, the fresh air system will be automatically started;

[0051] When the device state vector matrix detects that the TV is turned on and the ambient light is less than 50 lux, the backlight strip is brightened.

[0052] The present invention obtains multimodal data through the smart home device end, which can fully reflect the user's actual needs and preferences, avoids the information loss that may be caused by single modal data, and ensures the accuracy and response speed of the recommendation system; adopts modal feature extraction and cross-modal dynamic fusion algorithm, and adaptively weighted fusion of different modal data, which can adjust the weight in real time according to the changes in time and space dimensions, which not only improves the flexibility of the recommendation system, but also enhances its adaptability to complex environments; the device resource monitoring module monitors the CPU load and memory occupancy in real time, and dynamically adjusts the data calculation strategy according to the monitoring results, avoiding delays or performance bottlenecks caused by overload of computing resources, ensuring the stable operation of the system in different hardware environments, and further improving efficiency.

[0053] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0055] Figure 1 A flowchart of a method for actively recommending multimodal data based on cross-modal dynamic fusion and device resource awareness according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0056] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0057] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0058] Figure 1 A flowchart of a method for actively recommending multimodal data based on cross-modal dynamic fusion and device resource awareness according to an embodiment of the present disclosure is shown. The method includes:

[0059] S1, obtains multimodal data through smart home devices, including voice data, environmental sensor data, and status data of user-actively controlled devices;

[0060] S2, performing modal feature extraction on the multimodal data to generate a speech feature matrix, an environment probability matrix, and a device state vector matrix;

[0061] S3, using a cross-modal dynamic fusion algorithm, combining the time dimension and space dimension weights, to perform adaptive weighted fusion on the speech feature matrix, the environment probability matrix, and the device state vector matrix;

[0062] S4: Use the device resource monitoring module to monitor the CPU load and memory usage of the central control device in real time and dynamically adjust the data calculation strategy;

[0063] S5, based on the fused multimodal feature matrix, generates active recommendation instructions through user portrait modeling and knowledge graph association, driving smart home devices to perform adaptive adjustments.

[0064] According to an embodiment of the present invention, multimodal data is obtained through the smart home device end, which can fully reflect the user's actual needs and preferences, avoid the information loss that may be caused by single modal data, and ensure the accuracy and response speed of the recommendation system; by adopting modal feature extraction and cross-modal dynamic fusion algorithm, adaptive weighted fusion of different modal data can adjust the weight in real time according to changes in time and space dimensions, which not only improves the flexibility of the recommendation system, but also enhances its adaptability to complex environments; the device resource monitoring module monitors the CPU load and memory occupancy in real time, and dynamically adjusts the data calculation strategy according to the monitoring results, avoiding delays or performance bottlenecks caused by overload of computing resources, ensuring the stable operation of the system in different hardware environments, and further improving efficiency.

[0065] In some embodiments, the method for generating the speech feature matrix in S2 includes: converting the user speech into text data through speech recognition technology to obtain first business data; using a word embedding model to vectorize the first business data to generate a word vector sequence; using a Relu activation function to perform a nonlinear transformation on the word vector sequence to extract speech semantic features; dividing the time dimension into unit time periods, calculating the mean of the word vector in each time period, and constructing a time-semantic dual-dimensional matrix as a speech feature matrix.

[0066] In some embodiments, the method for generating the environmental probability matrix in S2 includes: obtaining indoor temperature and formaldehyde concentration data from security equipment or environmental sensors to obtain second business data; normalizing the second business data and extracting statistical features; using the Softmax function to calculate the reporting probability of each environmental indicator within a preset time period to generate an environmental probability matrix with probability values ​​as elements; when the environmental comfort level is standard, the matrix value with a higher probability is used as the recommended target; when the security or formaldehyde or non-comfort environment value is abnormal, the recommended value of the recommendation algorithm is triggered. According to an embodiment of the present invention, by directly obtaining the key environmental indicators of the security equipment or sensor, the processing of redundant data is avoided and the initial computing load is reduced; by normalizing the data, different environmental indicators are placed on the same numerical scale, the accuracy of subsequent feature extraction and probability calculation is improved, and the risk of algorithm misjudgment is reduced; by mapping multi-dimensional environmental indicators to probability distributions, anomalies are quickly identified through mathematical methods, replacing complex rule judgments, and reducing computational complexity.

[0067] In some embodiments, the method for generating the device state vector matrix in S2 includes: obtaining the device working state data actively controlled by the user to obtain the third business data; performing one-hot encoding and standardization on the device switch state and parameter setting value; converting the processed data into a vector of the parameter quantity dimension to construct the device state vector matrix; updating the matrix through a sliding window mechanism to retain the device state change trajectory of the most recent N time units. According to an embodiment of the present invention, by obtaining the device data actively controlled by the user, it is ensured that high-value information is processed and the overhead of redundant data processing is reduced; by one-hot encoding and standardizing the unified data format and dimension, the model convergence speed is improved and the computational complexity is reduced; by adapting the vector and matrix data structure to hardware acceleration, the processing speed is improved by using efficient linear algebra operations; by retaining only the recent state trajectory through the sliding window mechanism, the memory usage and computational complexity are reduced, while the ability to respond to real-time changes is enhanced.

[0068] In some embodiments, S3's cross-modal dynamic fusion algorithm includes: calculating the weight coefficients of voice, environment and device modalities based on the attention mechanism: attention_weights = softmax(MLP(concat(voice_emb,env_emb,device_emb))), weighted summing of multimodal features according to weights: fused_feature = Σ(attention_weights[i]*modality_emb[i]), combining the spatial dimension weight matrix, and performing spatial correlation correction on the fused features, where the weight of devices in the same room is 1, the weight of devices on the same floor is 0.25, and the rest are 0; using a dynamic time warping algorithm based on DTW to perform time series alignment on real-time voice data and sensor data. According to an embodiment of the present invention, the importance weights of voice, environment and device modalities are automatically learned through softmax (MLP), focusing on key information, suppressing irrelevant noise, and reducing redundant calculations; by weighted fusion of multimodal features according to weight coefficients, the core information of high-weight modalities is retained, avoiding the explosion of feature dimensions and increased computational complexity caused by direct splicing; fixed weights are assigned according to device locations, and spatial prior knowledge is used to constrain the fusion process, reducing interference from long-distance irrelevant devices and improving computational targeting; the dynamic time warping algorithm (DTW) is used to solve the problem of temporal misalignment between voice and sensor data, reducing feature matching errors caused by time asynchrony, and improving the accuracy of subsequent decisions.

[0069] In some embodiments, the device resource monitoring strategy of S4 includes: when it is detected that the CPU usage of the central control device is ≥90%, the calculation task is postponed for 5 minutes; if the resource conditions are still not met for 3 consecutive detection cycles, the calculation is forced to start and the monitoring cycle is reset; when a JVM insufficient memory warning is received, the main algorithm sends a pause instruction for 10 minutes to trigger the garbage collection mechanism; after the resources are restored, the delayed high-priority recommended tasks are executed first. According to an embodiment of the present invention, by postponing high-CPU tasks, performance degradation or crash caused by system overload is prevented, and the core functions are guaranteed to run stably; indefinite delays are avoided through a forced start mechanism, ensuring that key tasks are eventually executed and backlog risks are reduced; by pausing tasks and triggering garbage collection when memory is insufficient, resources are released and interruptions caused by frequent GC or OOM are reduced; after resource recovery, high-priority tasks are processed first, resource allocation is optimized, and the efficiency of key business response is improved.

[0070] In some embodiments, the user portrait modeling method of S5 includes: constructing voice interaction history features: extracting semantic vectors of user high-frequency commands through the Transformer encoder; constructing environmental behavior pattern features: using a temporal convolutional network (TCN) to analyze the temporal change rules of temperature and light; constructing device usage habit features: modeling device linkage relationships based on a graph neural network to generate a device operation frequency matrix; inputting three-dimensional features into a multimodal fusion layer, and associating user commands, device operation records, and environmental states through a knowledge graph. According to an embodiment of the present invention, the self-attention mechanism of the Transformer encoder is used to efficiently capture long-distance semantic dependencies, accurately extract user intentions, and reduce command parsing time; through the parallelized convolution structure and dilated convolution design of the TCN, while ensuring the accuracy of temporal feature extraction, the processing speed of temperature / light data is significantly improved; through the graph neural network, the nonlinear linkage relationship between devices is accurately described, and the message passing mechanism is used to efficiently generate representations, avoiding the complexity of manual rule design; through the knowledge graph, presetting entity relationships to provide prior topological constraints, accelerate cross-modal feature alignment, and prevent the combinatorial explosion problem when merging multi-source data.

[0071] In some embodiments, the knowledge graph association includes: mapping the user voice command "dim the lights" to a node, associating the lighting device node and the light sensor node; when the "a bit cold" command is detected, combining the temperature and humidity sensor data and the historical set temperature of the air conditioner to generate a recommended heating instruction; identifying cross-device linkage rules through a graph reasoning engine. According to an embodiment of the present invention, by mapping voice commands to nodes, a rapid conversion from natural language to structured data is achieved, and device actions are directly triggered through predefined associations between nodes, reducing the intermediate links of semantic parsing and command distribution; by combining real-time sensor data with historical behavior patterns, accurate operation suggestions are automatically generated through rules or algorithms, avoiding the cognitive and operational costs required for users to manually integrate environmental status and preferences; by utilizing the graph relationship network to efficiently express the logical dependencies between devices, and by using graph traversal technology to quickly identify potential paths for cross-device linkage, the computational complexity of dynamic rule matching in complex scenarios is reduced.

[0072] In some embodiments, smart home devices perform adaptive adjustments including: selecting a 2D / 3D visualization solution based on the device's graphics rendering capabilities; generating a floating window containing operation suggestions when a recommended instruction is triggered; optimizing the UI layout based on user feedback data, with high-frequency operation instructions prioritized for top display; and directly pushing scalar data captured by high-priority security and abnormal environment sensing, as well as wind and rain sensor probes, to user perception. According to an embodiment of the present invention, a 2D / 3D solution is selected based on hardware performance to avoid performance bottlenecks, ensure smooth operation, reduce rendering delays and resource consumption; proactively provide operation suggestions through a floating window, shorten the user's decision path, reduce the time spent manually searching for functions, and improve interaction efficiency; and based on user feedback, high-frequency instructions are placed at the top, reducing operation complexity, reducing click / browsing levels, directly adapting to user habits, and improving operation speed.

[0073] In some embodiments, when the "sleep mode" keyword is detected in the voice feature matrix, the lights are turned off and the air conditioner speed is lowered; when the probability of formaldehyde concentration exceeding the standard in the environmental probability matrix is ​​greater than 85%, the fresh air system is automatically activated; when the device state vector matrix detects that the TV is on and the ambient light is less than 50 lux, the background light strip is brightened. According to an embodiment of the present invention, preset voice command recognition directly associates device control, reducing manual operation steps and improving response speed; based on high-probability risk prediction, countermeasures are initiated in advance to avoid the problem from worsening and shorten risk handling delays; and environmental parameters are automatically optimized by integrating device status and sensor data, reducing the need for manual adjustment and ensuring the consistency of scene experience.

[0074] It should be noted that, for simplicity of description, the aforementioned method embodiments are presented as a series of combined actions. However, those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, as certain steps may be performed in a different order or simultaneously, according to the present disclosure. Furthermore, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present disclosure. In the technical solutions of the present disclosure, the acquisition, storage, and application of user personal information involved comply with relevant laws and regulations and do not violate public order and good morals. It should be understood that the various forms of the above-mentioned processes can be used, and steps can be reordered, added, or deleted. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure are achieved. This is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of protection of the present disclosure.

Claims

1. A multimodal data active recommendation method based on cross-modal dynamic fusion and device resource perception, characterized in that: include: S1, obtains multimodal data through smart home devices, including voice data, environmental sensor data, and status data of user-actively controlled devices; S2, performing modal feature extraction on the multimodal data to generate a speech feature matrix, an environment probability matrix, and a device state vector matrix; S3, using a cross-modal dynamic fusion algorithm, combining the time dimension and space dimension weights, to perform adaptive weighted fusion on the speech feature matrix, the environment probability matrix, and the device state vector matrix; S4: Use the device resource monitoring module to monitor the CPU load and memory usage of the central control device in real time and dynamically adjust the data calculation strategy; S5, based on the fused multimodal feature matrix, generates active recommendation instructions through user portrait modeling and knowledge graph association, driving smart home devices to perform adaptive adjustments.

2. The multimodal data active recommendation method based on cross-modal dynamic fusion and device resource perception according to claim 1 is characterized in that: The method for generating the speech feature matrix in S2 includes: Converting the user's voice into text data through voice recognition technology to obtain first business data; Performing vectorization processing on the first business data using a word embedding model to generate a word vector sequence; A Relu activation function is used to perform nonlinear transformation on the word vector sequence to extract speech semantic features; The time dimension is divided into unit time periods, the mean of the word vector in each time period is calculated, and a time-semantic dual-dimensional matrix is ​​constructed as the speech feature matrix.

3. The method for active recommendation of multimodal data based on cross-modal dynamic fusion and device resource perception according to claim 2 is characterized in that: The generation method of the environmental probability matrix in S2 includes: Acquire indoor temperature and formaldehyde concentration data from security equipment or environmental sensors to obtain second business data; Normalizing the second service data and extracting statistical features; The Softmax function is used to calculate the reporting probability of each environmental indicator within the preset time period, and an environmental probability matrix with probability values ​​as elements is generated; When the environmental comfort level is standard, the matrix value with a higher probability is used as the recommended target; when the security, formaldehyde or non-comfort environment value is abnormal, the recommended value of the recommendation algorithm is triggered.

4. The method for active recommendation of multimodal data based on cross-modal dynamic fusion and device resource perception according to claim 3 is characterized in that: The method for generating the device state vector matrix in S2 includes: Obtaining the working status data of the device actively controlled by the user to obtain the third service data; Perform one-hot encoding and standardization on the device switch status and parameter setting values; Convert the processed data into a vector of parameter quantity dimension and construct the device state vector matrix; The matrix is ​​updated through a sliding window mechanism to retain the device state change trajectory of the last N time units.

5. The method for active recommendation of multimodal data based on cross-modal dynamic fusion and device resource perception according to claim 4 is characterized in that: S3's cross-modal dynamic fusion algorithms include: Calculate the weight coefficients of voice, environment and device modality based on the attention mechanism: attention_weights=softmax(MLP(concat(voice_emb,env_emb,device_emb))) Perform weighted summation of multimodal features by weight: fused_feature=Σ(attention_weights[i]*modality_emb[i]) Combined with the spatial dimension weight matrix, the fusion features are spatially correlated, where the weight of devices in the same room is 1, the weight of devices on the same floor is 0.25, and the rest are 0; A dynamic time warping algorithm based on DTW is used to align the real-time voice data with the sensor data.

6. The method for active recommendation of multimodal data based on cross-modal dynamic fusion and device resource perception according to claim 5 is characterized in that: S4's device resource monitoring strategies include: When the CPU usage of the central control device is detected to be ≥90%, the calculation task will be postponed for 5 minutes; if the resource conditions are still not met after three consecutive detection cycles, the calculation will be forced to start and the monitoring cycle will be reset; When receiving a JVM low-memory warning, the main algorithm sends a pause instruction for 10 minutes to trigger the garbage collection mechanism; After resources are restored, delayed high-priority recommended tasks are executed first.

7. The method for active recommendation of multimodal data based on cross-modal dynamic fusion and device resource perception according to claim 6 is characterized in that: S5's user portrait modeling methods include: Constructing voice interaction history features: Using the Transformer encoder to extract semantic vectors of the user's high-frequency commands; Constructing environmental behavior pattern features: using a temporal convolutional network (TCN) to analyze the temporal changes in temperature and light; Build device usage habit features: Model device linkage relationships based on graph neural networks and generate a device operation frequency matrix; The three-dimensional features are input into the multimodal fusion layer, and user instructions, device operation records and environmental status are associated through the knowledge graph.

8. The method for active recommendation of multimodal data based on cross-modal dynamic fusion and device resource perception according to claim 7 is characterized in that: Knowledge graph associations include: Map the user voice command "dim the lights" to a node, and associate it with the lighting device node and the light sensor node; When a "a bit cold" command is detected, a temperature increase recommendation is generated based on the temperature and humidity sensor data and the historical set temperature of the air conditioner; Identify cross-device linkage rules through the graph inference engine.

9. The method for active recommendation of multimodal data based on cross-modal dynamic fusion and device resource perception according to claim 8, characterized in that: Smart home devices perform adaptive adjustments including: Choose a 2D / 3D visualization solution based on the device's graphics rendering capabilities; When the recommended instruction is triggered, a floating window containing action suggestions is generated; The UI layout is optimized based on user feedback data, and high-frequency operation instructions are prioritized for top display.

10. The method for active multimodal data recommendation based on cross-modal dynamic fusion and device resource perception according to claim 9, characterized in that: When the "sleep mode" keyword is detected in the voice feature matrix, the lights are turned off and the air conditioner speed is lowered; When the probability of formaldehyde concentration exceeding the standard in the environmental probability matrix is ​​greater than 85%, the fresh air system will be automatically started; When the device state vector matrix detects that the TV is turned on and the ambient light is less than 50 lux, the backlight strip is brightened.

Images