An intelligent home scene control method and system based on edge computing

Abstract

The application provides an edge computing-based smart home scene control method and system, and relates to the technical field of Internet of Things. By acquiring multi-modal data such as cameras, audio collectors and environmental sensors and converting them into tensor representations, a first data set is generated in combination with prior rules, and then clustering analysis is performed on the second text data to filter relevant information. Finally, multi-round reasoning is performed in a partitioned cache manner in a large language model deployed on the edge side, realizing accurate control of home devices. The first partition retains multi-modal core information, and the second partition can dynamically update user instructions or logs, so as to continuously focus on key information in a long sequence context and flexibly process new data. This method makes full use of lightweight technologies such as pruning and quantization to maintain real-time and efficient reasoning effect in a limited computing environment, improving the accuracy and response speed of multi-modal scene understanding and home control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things, in particular to an intelligent home scene control method and system based on edge computing. BACKGROUND

[0002] With the increasing number of cameras, audio collectors, and various sensors such as temperature and humidity, infrared, etc. in the home environment, traditional intelligent home systems can only perform shallow control on single or partial data sources, and it is difficult to consider real-time collection and fusion of multi-modal information. Moreover, when large-scale computing is performed in the cloud, there are problems such as high bandwidth occupation and high risk of privacy leakage. In recent years, edge computing technology provides the possibility for deploying efficient multi-modal processing and inference algorithms in local terminals, but in actual application, there are still challenges such as too large model size, long context forgetting, and insufficient attention to key scene information, which makes it difficult to fully meet the real-time linkage and accurate response needs of home multi-modal scenes. In view of the above problems, there is an urgent need for an intelligent home scene control method that can efficiently process multi-modal data on the edge side and continuously focus on key information, thereby effectively integrating image, audio, sensor data, and user instructions, achieving fast and flexible control of intelligent home devices and ensuring privacy security. SUMMARY

[0003] In view of the deficiencies of the prior art, the embodiments of the present application provide an intelligent home scene control method and system based on edge computing.

[0004] In a first aspect, the embodiments of the present application provide an intelligent home scene control method based on edge computing, comprising:

[0005] Obtaining multi-modal data from a target device, and converting the multi-modal data into a tensor representation through a pre-trained encoder;

[0006] Based on the tensor representation and prior rules, generating first text data including image and / or audio description, and combining the tensor representation and the first text data to form a first data set;

[0007] Obtaining a plurality of second text data, and performing clustering analysis on the plurality of second text data according to the characteristics of the first text data, to associate second text data related to the current home scene with the first data set; wherein the second text data includes user instructions, device logs, and scene configurations;

[0008] Storing the first data set and the second text data in different cache partitions of a large language model deployed in an edge computing device respectively, and interacting with the large language model to generate a first output result;

[0009] updating the second text data based on the first output result, and interacting with the large language model based on the first data set and the updated second text data to generate a second output result;

[0010] The second output result is used to control an intelligent home device.

[0011] As an optional implementation, the multi-modal data includes image data collected by a camera, audio data collected by an audio collector, and temperature and humidity or infrared information output by an environment sensor.

[0012] The conversion of the multi-modal data into a tensor representation by the pre-trained encoder includes:

[0013] The image data collected by the camera and the audio data collected by the audio collector are subjected to feature extraction by a light-weight convolutional self-attention hybrid network.

[0014] The temperature and humidity or infrared information output by the environment sensor is subjected to vectorization mapping and fusion processing with the features of the image data and the audio data.

[0015] In the features generated after the fusion processing, a pruning or quantized self-attention module is used to model global correlation of multi-modal features to obtain a tensor representation corresponding to the multi-modal data.

[0016] As an optional implementation, the light-weight convolutional self-attention hybrid network includes:

[0017] The image data is divided into image blocks, and the feature vectors extracted from the audio data are inserted into the sequence of the image blocks, and the image blocks and the audio vectors are jointly modeled by fusion convolution operation and multi-head self-attention mechanism.

[0018] As an optional implementation, the pruning or quantized self-attention module is used to model global correlation of multi-modal features to obtain a tensor representation corresponding to the multi-modal data includes:

[0019] The home scene context information and the environment sensor state are obtained.

[0020] According to the home scene context information and the environment sensor state, a priority attention head in a target time period is determined, and a pruning or closing operation is performed on a non-priority attention head in the self-attention module.

[0021] The priority attention head is retained with a preset high-precision parameter weight for processing key information appearing in night audio features or in daytime image features.

[0022] The pruning or closing operation is completed by the attention head and the attention head of the priority concern, and the global correlation modeling of the multi-modal feature is outputted.

[0023] As an optional implementation, the pruning or quantization of the self-attention module for global correlation modeling of multi-modal features further includes:

[0024] Collect and record the interaction frequency of user instructions within a preset period;

[0025] In response to the interaction frequency of the user instruction being greater than or equal to a first threshold, the quantization precision of the self-attention module is set to a first bit width; wherein the first bit width is used to control the processing precision of audio and / or image features;

[0026] In response to the interaction frequency of the user instruction being less than a second threshold, the quantization precision of the self-attention module is switched to a preset low bit width.

[0027] As an optional implementation, the forming of the first data set includes:

[0028] Obtain home scene context information and user current behavior data;

[0029] According to the home scene context information and the user current behavior data, combined with the time period feature and the device state, dynamically adjust the content and precision of the generated image description and audio description;

[0030] Based on the adjusted description text, key information is extracted from the context based on prior rules to generate first text data containing home scene state;

[0031] The generated first text data and the tensor representation are spliced to form a first data set.

[0032] As an optional implementation, the obtaining of the plurality of second text data and the clustering analysis of the plurality of second text data according to the features of the first text data to associate the second text data related to the current home scene with the first data set includes:

[0033] Obtain a plurality of second text data from user instructions, device logs and scene configurations, and perform word segmentation preprocessing on the second text data;

[0034] Based on the key fields or context information contained in the first text data, a similarity measure function is constructed; the similarity measure function is used to measure the association degree between the second text data and the first text data;

[0035] performing clustering analysis on the plurality of second text data using the similarity measure function to generate a plurality of clustering clusters;

[0036] associating the clustering cluster of the second text data with the similarity greater than or equal to the preset threshold with the first data set to form a set of text information matching the current home scene.

[0037] As an optional implementation, the generating a first output result includes:

[0038] storing the tensor representation contained in the first data set and the corresponding image and / or audio description text in a first partition of the large language model cache to reserve multi-modal key information;

[0039] storing the second text data in a second partition of the large language model cache and indexing or labeling the second text data based on the type of the second text data;

[0040] simultaneously reading data in the first partition and the second partition as input to the large language model for inference to obtain the first output result;

[0041] in response to the first output result, updating or replacing part of the text data in the second partition, and continuing to maintain the tensor representation and the corresponding description text unchanged in the first partition.

[0042] As an optional implementation, the updating the second text data based on the first output result, and then interacting with the large language model based on the first data set and the updated second text data to generate a second output result includes:

[0043] parsing key information indicating replacement, insertion, and deletion operations on the second text data from the first output result;

[0044] updating the corresponding second text data according to the key information in the second partition of the large language model cache;

[0045] maintaining the tensor representation and the image and / or audio description text contained in the first partition unchanged;

[0046] re-reading data in the first partition and the updated second partition as input to the large language model for inference to obtain the second output result.

[0047] In a second aspect, the embodiments of the present application provide an intelligent home scene control system based on edge computing, comprising:

[0048] The acquisition unit is configured to acquire multi-modal data from a target device and convert the multi-modal data into a tensor representation through a pre-trained encoder;

[0049] The first processing unit is configured to generate first text data including image and / or audio descriptions based on the tensor representation and prior rules, and combine the tensor representation and the first text data to form a first data set;

[0050] The second processing unit is configured to acquire a plurality of second text data and perform cluster analysis on the plurality of second text data according to features of the first text data to associate second text data related to a current home scene with the first data set; wherein the second text data includes user instructions, device logs, and scene configurations;

[0051] The generation unit is configured to store the first data set and the second text data in different cache partitions of a large language model deployed in an edge computing device respectively, and interact with the large language model to generate a first output result;

[0052] The adjustment unit is configured to update the second text data based on the first output result, and interact with the large language model based on the first data set and the updated second text data to generate a second output result; wherein the second output result is used to control smart home devices.

[0053] Compared with the prior art, the present application converts multi-modal data into a tensor representation on an edge computing device and generates a first data set with prior rules, then filters relevant information through cluster analysis on a plurality of second text data, and realizes efficient multi-round reasoning and flexible updating with partitioned cache management of a large language model. On the one hand, the fixed first partition can continuously focus on core scene information, avoiding forgetting in long sequence reasoning; on the other hand, the second partition that can be updated allows quick insertion, replacement or deletion of text data when user instructions or logs change, thereby reducing unnecessary repeated calculations and timely adapting to environmental switching. Through the joint action of multi-modal encoding, attention pruning / quantization and scene prior rules, the present application can accurately understand multi-source information such as video, audio and sensors in a home scene, and maintain low-latency reasoning on resource-limited edge devices, ultimately realizing efficient control and personalized response of home devices. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart of an intelligent home scene control method based on edge computing provided by an embodiment of the present application;

[0055] Figure 2A method flowchart for converting the multi-modal data into a tensor representation by a pre-trained encoder is provided for the embodiments of the present application.

[0056] Figure 3 A schematic diagram of an intelligent home scene control system based on edge computing is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments of the present application.

[0058] Referring to Figure 1 A flowchart of an intelligent home scene control method based on edge computing is provided for the embodiments of the present application, including steps S101-S105, wherein:

[0059] S101: acquiring multi-modal data from a target device, and converting the multi-modal data into a tensor representation by a pre-trained encoder;

[0060] S102: generating first text data including image and / or audio description based on the tensor representation and prior rules, and combining the tensor representation and the first text data to form a first data set;

[0061] S103: acquiring a plurality of second text data, and performing clustering analysis on the plurality of second text data according to the features of the first text data, to associate second text data related to the current home scene with the first data set; wherein the second text data includes user instructions, device logs and scene configurations;

[0062] S104: storing the first data set and the second text data in different cache partitions of a large language model deployed in an edge computing device respectively, and interacting with the large language model to generate a first output result;

[0063] S105: updating the second text data based on the first output result, and interacting with the large language model based on the first data set and the updated second text data to generate a second output result; wherein the second output result is used to control intelligent home devices.

[0064] In the present application, through the combination of multi-modal data acquisition and pre-training encoder, it is possible to fuse multi-source information such as camera image, audio acquisition data and environmental sensor acquisition data in real time on the edge computing device; then, based on the prior rules, the preliminary analysis is carried out on the encoded tensor representation, and the description text summarizing the current home scene is generated, and the first data set is constructed combining the description text and the tensor representation. At the same time, the second text data is also collected from user instructions, device logs and scene configurations, and the clustering analysis is carried out on these text data under the guidance of the first data set, and only the information strongly related to the actual scene is associated with the first data set. Then, the first data set and the clustered second text data are respectively stored in different cache partitions of the large language model in the edge computing device, and the large language model is used to infer the content of each partition. If it is found in the inference output that part of the second text data needs to be updated, the system will perform the corresponding modification and input the model again with the first data set to generate a more accurate second output result, and finally drive the smart home device to perform the corresponding control operation. The design of multi-modal data fusion, partitioned storage and dynamic update can continuously focus on key information and reduce the computational burden in the edge computing environment, thereby significantly improving the efficiency and accuracy of intelligent control of complex home scenes.

[0065] Among them, the key information refers to the characteristic data or semantic content that has a significant impact on the subsequent large language model inference or home scene control decision after preliminary analysis of multi-modal data and its textual description. Specifically, these key information is related to the core context of the current home environment or user demand, including but not limited to suspicious audio features in night mode, significant image activity areas in daytime scenes, and text instructions or log contents that are highly matched with the first data set through prior rules or similarity judgment, etc. If some image blocks, audio segments or text records are identified as key information, they will be stored in the first partition of the large language model cache or marked with a high priority to ensure that they are continuously retained and can be repeatedly used in the multi-round inference process.

[0066] In this way, not only can the system avoid forgetting the most valuable multi-modal features or context text in a long sequence, but also can prevent irrelevant or low-value information from interfering with the inference process in the edge device environment with limited computing power, thereby achieving more accurate and efficient home scene control.

[0067] For the above S101:

[0068] In specific implementations, a multi-channel data acquisition method can be used, for example, image frames are captured from the camera every fixed interval, acoustic waveforms are obtained from the audio collector, and numerical information is read from the temperature and humidity or infrared sensor. The pre-trained encoder can process images, audio, and sensor values into a unified dimensional vector or tensor form based on a lightweight convolution-self-attention hybrid network (or other edge-computing-adaptive models).

[0069] By directly completing the tensorization of multi-modal data on the local edge device, the scene information can be fully retained when inputting the large language model subsequently, and the bandwidth and latency problems caused by long sequence input can be reduced.

[0070] For the above S102:

[0071] The prior rules can include night / day, user location, family usage habits, etc. The focus of attention required for images / audio in different scenarios is different.

[0072] In specific implementations, according to the pre-trained scene recognition algorithm or key frame detection result, the image tensor is parsed into a scene label (such as "two people are active in the living room"), the audio tensor is parsed into a sound event (such as "there is a slight conversation sound"), and it is converted into text data. The above text description and tensor representation are combined into a whole data structure to save the key semantics and features of multi-modal.

[0073] In this way, through the combination of prior rules and multi-modal tensor representation, scene-related summary text can be quickly generated, which not only guarantees readability, but also provides clear context reference for clustering analysis and large language model reasoning in the subsequent stage.

[0074] In specific implementations, the prior rules can be flexibly extended or combined according to different family environments, user preferences, and daily habits to adapt to the needs of multi-modal perception and reasoning.

[0075] In specific applications, the prior rules can be stored in the local database or file of the edge computing device, and matched or retrieved for multi-modal data during preprocessing and scene analysis.

[0076] For example, in a "nighttime scenario," if the illuminance sensor detects that the ambient brightness is below a certain threshold (e.g., 50 lux) and the time has passed 10 PM, the system determines that it is in "night mode." In this case, prior rules can assign higher weight to audio data to capture possible subtle sounds or voice commands; simultaneously, the granularity of image segmentation can be reduced to save computational resources. If the system detects that the user is sleeping or there is no activity in the room, prior rules may further require highlighting the keyword "nighttime monitoring" in the text description and favor matching with security logs or abnormal alarm commands during cluster analysis.

[0077] In the "daytime scene," if the environmental sensor detects light levels above 200 lux and the user is moving frequently, a priori rule can be triggered to refine the image segmentation process (e.g., preserving the complete channels of the convolutional layer), while the audio can be weighted normally. If the time period information shows 7:00 AM to 9:00 AM, the priori rule can also prompt the system to add text descriptions such as "may be in a busy period" or "preparing breakfast" to the first text data, and pay more attention to kitchen equipment logs or breakfast automation configuration scripts during cluster analysis to quickly link the corresponding home appliances.

[0078] Regarding user behavior, if the sensor detects movement within the infrared detection area, or if obvious conversations are captured in the audio, prior rules can instruct the system to include "activity detection" or "conversational interaction" as scene keywords in the text description. This allows for priority clustering of corresponding instructions or home logs during large language model inference. Conversely, if the system detects prolonged periods of inactivity and infrared readings approaching zero, prior rules can be triggered to indicate an "empty scene." This makes the security mode a reference text during the inference phase, helping the system decide whether to switch to "away from home" or "enhanced security" mode.

[0079] In addition to day / night cycles and user behavior, prior rules can also be extended to other household usage habits. For example, increasing air conditioner checks when temperatures are high in summer, paying attention to heating equipment operation when it's cold in winter, or based on family members' daily routines (such as emphasizing nighttime lighting instructions for elderly people who frequently get up at night). These rules can be closely integrated with the multimodal perception stage of this application: once "temperature display value is too high" is detected in the image embedding or "abnormal equipment operating noise" is recognized in the audio, the prior rules can be combined to add statements such as "air conditioner maintenance suggestions" or "equipment noise alarm" to the first text data to assist cluster analysis and large language model reasoning in timely attention to relevant text data (such as air conditioner logs, maintenance documents, etc.).

[0080] Regarding the above S103:

[0081] In practice, the second text data may include control commands issued by the user via voice / text, device logs (such as air conditioner operation records, security alarm information, etc.), and scene configurations (such as home automation scripts and schedules).

[0082] Furthermore, the text can be segmented and vectorized, and then similarity comparisons or topic analyses can be performed using keywords or feature vectors related to the scene descriptions in the first dataset. Finally, the most relevant texts are divided into several clusters. The clusters of texts with relevance higher than a threshold are associated with the first dataset to ensure that only the text information most needed in the current scene is introduced during subsequent inference.

[0083] This avoids loading a large amount of irrelevant or noisy text during large language model inference, improving inference efficiency and reducing interference, ensuring that the system can respond to scene changes in real time in a home environment.

[0084] Regarding S104 above:

[0085] In practice, the first dataset (including multimodal tensors and core descriptions) is fixed in the first partition to form a fixed context in the long sequence interaction of the large language model; while the associated second text data is placed in the second partition so that it can be dynamically replaced or updated according to the needs of inference.

[0086] Furthermore, by deploying a lightweight large language model or hierarchical model structure at the edge, reasoning on the complete context of "first partition + second partition" is achieved, and the initial output result (i.e. the first output result) is obtained.

[0087] In this way, through partitioned management, preliminary reasoning of home scenarios can be completed without reloading or losing key multimodal information, and outputs such as user command parsing and device operation recommendations can be provided.

[0088] Since home smart gateways or microservers often have limited computing power (e.g., only low-power CPUs and a small number of NPU / TPU acceleration resources), directly using large-scale general-purpose pre-trained models (e.g., with billions or tens of billions of parameters) is difficult to meet the latency requirements of inference locally. Therefore, it is necessary to perform lightweight processing of the model or choose a miniaturized version. The following lists several common model and specific strategies for pruning, distillation, or quantization to further illustrate the feasibility and key points of performing large language model inference in home edge devices.

[0089] In practical implementation, the core network structure can be extracted from common large-scale language models such as GPT, LLaMA, etc., and the number of parameters can be reduced and the inference overhead can be lowered through optimization.

[0090] For example, in models with many stacked Transformer layers, some mid-to-late stage layers can be removed or the number of multi-head attention layers can be reduced. For instance, reducing the original 12 layers to 6 layers can maintain the basic ability to understand language sequences, but sacrifice some complex reasoning capabilities in exchange for deployability at the edge.

[0091] For example, for multi-head self-attention structures, if the evaluation finds that some attention heads contribute little in most home scenarios, some attention heads or channel convolution kernels can be removed using pruning tools (such as AST, structured sparsity libraries, etc.) to further reduce the parameter size.

[0092] For example, low-precision quantization such as int8 or int4 can be used to store and compute model weights and activation values. In home scenarios, int8 quantization can typically reduce memory usage and computing power requirements by one to two orders of magnitude without significantly impairing text understanding and multimodal fusion performance. If the hardware NPU / TPU supports int8 operators or int4 acceleration, inference throughput and parallelism can be further improved.

[0093] From a hardware compatibility perspective, if the edge device used contains a specific neural network accelerator (such as NVIDIA Jetson, Arm Ethos-N, or Google Edge TPU), support for quantization inference with precisions such as int8 / float16 can usually be achieved in the SDK that comes with these hardware devices.

[0094] In general-purpose CPU environments, layered compilation or automated quantization can be performed using tools such as Intel OpenVINO, TensorRT, or ARM NN SDK to convert the trimmed Transformer into a model file suitable for local execution (such as ONNX or INT8 weights).

[0095] To facilitate adaptation to the multi-round inference mechanism of this application, it is also necessary to retain the read and write interfaces for cache partitions (such as KV cache) and to perform appropriate API encapsulation of the model forward computation process so as to quickly concatenate and infer based on the input tokens of different partitions.

[0096] For example, on a real-world home gateway, a small model of a distilled version of GPT or LLaMA-7B can be selected. After the aforementioned layer reduction and quantization, it can be scaled down to within 300-400 million parameters, ensuring that memory usage does not exceed 1-2GB. Combined with the KV cache read / write interface, a one-time adaptation is performed for the partition interaction process. With this feasible implementation, the system can complete multimodal data parsing and text-based linked reasoning in a home scenario without cloud computing power, relying solely on local devices to achieve efficient real-time performance and privacy protection.

[0097] Regarding the above S105:

[0098] In practice, if it is found during the reasoning process that it is necessary to supplement some device logs, correct user commands, or insert new commands, the relevant text content will be replaced, inserted, or deleted in the second partition.

[0099] Furthermore, keeping the first partition unchanged, the updated second text data and the first dataset are input again into the large language model to obtain a more accurate or context-consistent second output. This output may include specific operation commands (such as turning devices on / off, adjusting temperature, switching scene modes, etc.) or provide predictive prompts (such as reminding users that the security system is in night mode), thereby enabling real-time control of the smart home system.

[0100] In this way, by utilizing the mechanism of multi-round interaction and partition update, the edge can flexibly respond to new user needs or changes in scenarios without the need for large-scale replacement of the first dataset, thereby reducing redundant calculations, improving response speed, and enhancing the accuracy and real-time performance of overall control.

[0101] As an optional implementation, the multimodal data includes: image data acquired by a camera, audio data acquired by an audio acquisition device, and temperature, humidity, or infrared information output by an environmental sensor.

[0102] Please see Figure 2 The flowchart of a method for converting multimodal data into tensor representation using a pre-trained encoder, provided in this application embodiment, includes steps S201 to S203, wherein:

[0103] S201: Feature extraction of image data from a camera and audio data from an audio acquisition device based on a lightweight convolutional self-attention hybrid network;

[0104] S202: Vectorize and map the temperature, humidity or infrared information output by the environmental sensor, and fuse it with the features of the image data and audio data;

[0105] S203: In the features generated after the fusion process, a self-attention module with pruning or quantization is used to perform global correlation modeling on the multimodal features to obtain the tensor representation corresponding to the multimodal data.

[0106] In an optional implementation, the multimodal data mainly comes from images captured by a camera, audio collected by an audio acquisition device, and temperature, humidity, or infrared values ​​output by environmental sensors. Specifically, during image acquisition, the edge computing device can obtain the real-time frame sequence provided by the camera via USB or network interface, and perform normalization and downsampling processing on key frames, such as scaling the image to 224×224 or a resolution more suitable for lightweight network inference; the audio signal is read in PCM or AAC format, and in some scenarios, short-time Fourier transform or Mel-frequency spectrum analysis can be performed to generate a fixed-dimensional audio feature vector.

[0107] Meanwhile, environmental sensors periodically report temperature, humidity, or infrared detection results via various methods (such as I2C or UART communication). The edge device converts this sensor data into small-scale vector representations (e.g., 16-dimensional or 32-dimensional) for subsequent fusion with image and audio features. The pre-trained encoder preferably employs a lightweight combination of convolutional and self-attention modules to fully extract multimodal information even in resource-constrained edge devices. Data from the camera and audio acquisition unit first passes through convolutional layers to extract local features, then undergoes global correlation learning through self-attention layers to obtain the fused first- or second-order embedding.

[0108] At the same time, the temperature, humidity or infrared information output by the environmental sensor will be mapped into a low-dimensional vector in the designated channel of the encoder, and then spliced ​​or weighted with the image and audio embeddings to form a unified multimodal feature representation.

[0109] In addition, in order to maintain efficient inference while taking into account real-time performance and accuracy in the scenario, pruning or quantization techniques can be further adopted when the self-attention module performs global correlation modeling of multimodal features: on the one hand, depending on the computing power of the edge computing device, some attention heads or channels can be structurally pruned to remove redundant weights.

[0110] On the other hand, key attention operations are quantized with low bit depth (such as INT8 or INT4) to reduce computational power and memory consumption. Through multimodal fusion and attention pruning / quantization operations of this lightweight network, a large-scale tensor representation for subsequent scene analysis or inference is finally obtained. This representation not only combines the spatial-temporal features of images and audio but also retains key numerical information from environmental sensors, thereby improving the overall understanding of multimodal home scenarios.

[0111] Furthermore, how can we efficiently model images and audio simultaneously on resource-constrained edge devices, and ensure that audio information is fully expressed within the spatial structure of image features? Conventional approaches often process images and audio separately, or simply add feature vectors together. This makes it difficult to achieve deep fusion of image patches and audio vectors within a single network structure, which can lead to insufficient attention to spatiotemporal resolution or audio features, thereby reducing the overall multimodal understanding performance.

[0112] As an optional implementation, the lightweight convolutional self-attention hybrid network includes:

[0113] The image data is divided into image blocks, and the feature vectors extracted from the audio data are inserted into the sequence of the image blocks. The image blocks and audio vectors are jointly modeled by fusing convolution operations and a multi-head self-attention mechanism.

[0114] In the specific implementation process, the image data can be divided into several image blocks, each representing a smaller pixel region or feature block, and arranged in the order of scanning or segmentation in time sequence; then, after the audio data is processed by short-time Fourier transform or Mel spectrum processing, the audio feature vector is extracted, and then this audio feature vector is inserted into certain positions in the image block sequence to achieve a unified representation of image and audio.

[0115] For example, an audio feature vector can be inserted after every few image blocks to ensure that the network learns the spatial structure of the images while also paying attention to important information in the temporal or frequency domains of the audio. Next, by fusing convolutional operations and a multi-head self-attention mechanism, the image blocks and audio vectors in the mixed sequence are jointly modeled: the convolutional operation provides feature aggregation for local image regions or adjacent audio frames, while the multi-head self-attention mechanism captures cross-modal relationships between image blocks and between image blocks and audio features globally, thus obtaining a multimodal representation that is compatible with spatial, temporal, and audio elements at the network output.

[0116] In this way, image segmentation ensures that the ability to recognize local details is still preserved under a lightweight network structure. On the other hand, inserting audio feature vectors into the image segment sequence enables the network to process image and audio information simultaneously, thereby avoiding the repeated construction of multiple models at the edge or the forced splicing of features in the later stage. In addition, the multi-head self-attention mechanism can not only extract the semantic relationships within the image segments, but also understand the fusion relationship between audio vectors and local image features, enhancing the overall cognitive ability of multimodal scenes. Through this joint modeling approach, this application can achieve efficient collaborative processing of images and audio on resource-constrained home edge computing devices and improve the accuracy of scene understanding and reasoning.

[0117] As an optional implementation, the step of using a pruning or quantized self-attention module to perform global correlation modeling on multimodal features to obtain the tensor representation corresponding to the multimodal data includes:

[0118] Obtain contextual information about the home scene and the status of environmental sensors;

[0119] Based on the home scene context information and the environmental sensor status, determine the attention heads that are given priority within the target time period, and perform pruning or shutdown operations on the non-priority attention heads in the self-attention module.

[0120] The attention head that is given priority retains a preset high-precision parameter weight for processing key information that appears in nighttime audio features or daytime image features;

[0121] The attention head that has undergone the pruning or closing operation and the attention head that receives priority attention jointly complete the global correlation modeling of multimodal features, and output the pruned tensor representation for subsequent inference.

[0122] When performing global correlation modeling of multimodal features, a technical challenge arises in edge computing devices: it is difficult to simultaneously achieve high accuracy and real-time performance. This is especially true in scenarios with significant differences in data distribution between night and day, audio and images, and different sensors. Applying the same level of accuracy to all attention points can lead to redundant computational overhead in environments with limited computing power. It can also consume excessive resources on non-critical modal information, thereby reducing the overall efficiency and real-time performance of the system.

[0123] In practical implementation, edge devices can first determine the most important modalities in the current multimodal data based on the monitored home scene context information (such as whether the current time is night or day) and the status values ​​output by environmental sensors (such as whether temperature, humidity, or infrared sensing exceeds thresholds). For example, in a nighttime scene, audio features are given priority to detect potential abnormal sounds or voice commands; during the daytime, more emphasis is placed on fine-grained processing of image features to identify family member activities or environmental changes. Subsequently, in the self-attention module, attention heads that are closely related to the current scene are given priority and have high-precision parameter weights, such as using higher bit widths or less quantization; while attention heads that are not given priority are pruned (i.e., some redundant channels or weights are removed) or directly turned off to reduce computation and save memory resources. Through this differentiated retention strategy, the network can maintain sufficient fine-grained analysis capabilities on key modalities while saving computational power on currently unimportant modalities, ensuring overall inference speed.

[0124] For example, in a nighttime security scenario, at night, environmental sensors detect extremely low illumination, and edge devices determine that the user has entered a resting state, naturally shifting the monitoring focus to audio features (such as doorbells, breaking sounds, or voices). At this time, based on the nighttime scene context, the system retains audio-related attention heads in a high-precision parameter weight state, such as using 8-bit or higher precision quantization, to ensure sensitivity to audio details. Simultaneously, it performs pruning or disables image-related attention heads, thereby reducing the impact of noise in nighttime image data and its computational burden. If a suspicious sound is detected, the audio attention head can accurately distinguish the sound, outputting a preliminary judgment result (such as outdoor banging sounds or suspicious voices), and prioritizing the audio text representation in subsequent inference.

[0125] This demonstrates that the differentiated attention strategy achieved by using pruning or quantization of self-attention modules can not only reduce the overall computational load at home edge devices with limited computing power, but also ensure in-depth analysis of key information (audio modalities at night).

[0126] The tensor representation obtained by performing global correlation modeling on multimodal features retains the most important modal information in a specific scenario, while minimizing resource consumption on unimportant or rare attention heads. Ultimately, it can be input into subsequent models or inference processes to output results that best fit the current scenario requirements, thereby improving the accuracy and response speed of nighttime security or daytime family activity detection.

[0127] As an optional implementation, the method of using a pruned or quantized self-attention module to perform global correlation modeling of multimodal features further includes:

[0128] Collect and record the frequency of user command interactions within a preset time period;

[0129] In response to the user instruction's interaction frequency being greater than or equal to a first threshold, the quantization precision of the self-attention module is set to a first bit width; wherein, the first bit width is used to control the processing precision of audio and / or image features;

[0130] If the frequency of interaction with the user instruction is less than a second threshold, the quantization precision of the self-attention module is switched to a preset low bit width.

[0131] In this optional implementation, it is necessary to balance the fine processing of multimodal features such as audio and images with real-time response speed in a home environment with limited computing power. Especially when users frequently send voice or text commands in a short period of time, keeping all attention in a high-precision quantization state may lead to excessive load on edge devices; however, if a low bit width is used uniformly, it is easy to lose accuracy in the recognition of audio or image features.

[0132] To address this technical challenge, a strategy is introduced into the self-attention module to dynamically adjust the quantization accuracy based on the frequency of user-instructed interactions.

[0133] In practical implementation, the frequency of user commands within a preset time period (e.g., the past 30 or 60 seconds) is continuously collected and recorded in home use scenarios. If the frequency of this interaction is greater than or equal to a first threshold (e.g., more than N commands), it means that the user is frequently issuing voice or text commands. To ensure high-fidelity processing of key modal signals, the system will automatically switch the quantization precision of the self-attention module to the first bit width (e.g., 8 bits or less with cropping / quantization loss), thereby improving the detail retention of audio and / or image features.

[0134] Conversely, when the frequency of interaction is less than the second threshold, it indicates that there are few user instructions or the system is in a low-interaction state. At this time, the quantization precision of the self-attention module can be switched to a lower bit width (such as 4 bits) to significantly reduce the computing power requirement and memory usage of the inference process.

[0135] This allows for more precise processing of multimodal features related to user commands during busy periods, ensuring effective recognition of audio or images, while significantly reducing computational load and redundant overhead during idle periods, enabling the entire self-attention network to operate more stably and economically at the edge.

[0136] As an optional implementation, forming the first dataset includes:

[0137] Obtain contextual information about the home environment and current user behavior data;

[0138] Based on the home scene context information and the user's current behavior data, combined with time period characteristics and device status, the content and accuracy of the generated image and audio descriptions are dynamically adjusted.

[0139] Based on the adjusted descriptive text, key information is extracted from the context using prior rules to generate first text data containing the state of the home scene.

[0140] The generated first text data is concatenated with the tensor representation to form the first dataset.

[0141] In this optional embodiment, the technical problem addressed is how to enable the first dataset to flexibly adapt to dynamic changes in the home environment, including the home environment at different times of day and frequent switching of the user's current behavioral state. For example, in the morning, the user may move around continuously and issue short commands, while environmental sensors simultaneously report increased illuminance and decreased humidity; while at night, the user stops moving or sends drastically different commands. This requires adaptive adjustments to the image and audio descriptions so that the first text data accurately reflects the current state of the home environment.

[0142] In practice, contextual information about the home environment (such as night or day, security mode or leisure mode) can be obtained and combined with the user's current behavior data (such as whether the user is cooking, sleeping, or watching TV). Based on these contextual elements, time period characteristics (such as peak hours, low-noise periods at night), and device status (such as air conditioning being on, lights being dimmed), the generated image and audio descriptions can be dynamically adjusted.

[0143] For example, when a user is cooking in the kitchen, the system can pay more attention to the sounds of "chopping vegetables" or "boiling" in the audio, and describe the living room camera image with lower precision; if it is nighttime and the user's behavior is "going to bed", it can reduce its attention to changes in the image and focus more on marking potential abnormal noises or doorbell sounds.

[0144] After this adjustment is completed, the system will use these modified scene description texts in conjunction with prior rules to extract information containing key points of the home scene (such as "kitchen activities" and "nighttime silent mode") from the context, thereby generating the first text data containing the state of the home scene.

[0145] Finally, the text data is concatenated or merged with the previously obtained multimodal tensor representation to construct the first dataset that can accurately represent the current scene environment and user activities, so as to continuously provide the system with the latest and most accurate home context references in subsequent cache partition management and multi-round large language model inference.

[0146] For example, consider a cooking session at night: the sensor detects that the kitchen lights are on, the time is after 10 p.m., the user's current behavior is identified as "obvious activity in the kitchen", at this time the audio collector captures more chopping sounds, and the system emphasizes keywords such as "kitchen activity" and "increased audio intensity" in the description text;

[0147] On the other hand, if the living room camera footage does not show significant changes, it is marked as a low-priority description, while the prior rules determine that "nighttime kitchen use" may be related to subsequent security or fire alarm monitoring.

[0148] In this context, text related to kitchen activities will be highlighted and combined with image and audio tensor representations to form the first dataset. This ensures that subsequent reasoning processes can correctly focus on the key scenario of nighttime cooking, thereby providing more targeted intelligent suggestions or security reminders.

[0149] In this way, by dynamically adjusting the content and accuracy of image and audio descriptions in conjunction with time periods, device status, and user behavior information when forming the first dataset, the system not only strengthens its grasp of key points of the scene, but also reduces the processing burden of unnecessary information at the edge, thereby improving the overall recognition and reasoning efficiency.

[0150] As an optional implementation, the step of acquiring multiple second text data and performing cluster analysis on the multiple second text data based on the characteristics of the first text data to associate the second text data related to the current home scene with the first dataset includes:

[0151] Acquire multiple second text data from user commands, device logs, and scene configurations, and perform word segmentation preprocessing on the second text data;

[0152] Based on the key fields or contextual information contained in the first text data, a similarity measurement function is constructed; the similarity measurement function is used to measure the degree of correlation between the second text data and the first text data.

[0153] The similarity metric function is used to perform cluster analysis on the multiple second text data to generate multiple clusters;

[0154] Clusters of the second text data with a similarity greater than or equal to a preset threshold are associated with the first dataset to form a set of text information that matches the current home scene.

[0155] In this optional implementation, it is necessary to quickly filter out content closely related to the first dataset from a large amount of text data from diverse sources to avoid introducing redundant or irrelevant information in subsequent reasoning processes. If different types of text, such as user instructions, device logs, and scene configurations, are not effectively distinguished and aggregated, the system may carry a large amount of redundant data during the large language model input stage, resulting in limitations on both the accuracy and efficiency of the reasoning results.

[0156] In practice, second text data is first obtained from multiple sources such as user commands, device logs, and scene configurations, and then preprocessed by word segmentation.

[0157] For example, text can be broken down into words using Chinese word segmentation tools or English word segmentation libraries, or vectorized using methods such as BERTembedding. Then, based on the key fields or contextual information contained in the first text data, a similarity metric function is constructed to measure the correlation between the second and first text data. These key fields can be important prompts reflecting the current home environment, such as "night mode" or "kitchen activity."

[0158] The system uses a similarity metric function to perform cluster analysis on all second text data, dividing the text content into several clusters and calculating the overall similarity between each cluster and the first text data. When a cluster with a similarity greater than or equal to a preset threshold is considered to be highly matched with the current home scene, the system will associate the text within these clusters with the first dataset to form a set of text information closely aligned with the needs of the current scene.

[0159] In this way, when faced with massive amounts of logs or commands in a home setting, the scenario description information of the first text data can be used as an index benchmark to efficiently filter and aggregate text that truly matches the current home situation, significantly reducing the amount of irrelevant input to the large language model. This not only improves inference accuracy but also effectively reduces the overhead of processing and storing large amounts of irrelevant text on edge computing devices, which has significant practical value for real-time response and resource-constrained home environments.

[0160] For example, in the process of clustering the second text data with the first text data, those skilled in the art can freely choose existing text similarity calculation methods according to their own needs, such as Bag-of-Words, TF-IDF, Word2Vec or BERT vector representations, and then combine them with K-means, DBSCAN or other common clustering algorithms.

[0161] In practical applications, texts such as user commands, device logs, and scene configurations can first be segmented into words or sentences and mapped to corresponding vector representations. Taking TF-IDF as an example, each text is represented as a vector containing feature words and their weighted frequencies in the document. If deep representation methods such as Word2Vec / BERT Embedding are used, the entire text or the segments after word segmentation can be projected into a vector space of hundreds of dimensions. Subsequently, to measure the correlation between a second text data and the first text data (which may have already been vectorized), a cosine similarity function can be used, with the vectors of the two texts denoted as v1 and v2 respectively. The calculation formula is: cos(θ)=(v1·v2) / (||v1||×||v2||).

[0162] Here, "·" represents the dot product operation, and "||v||" represents the vector norm. The closer this value is to 1, the more semantically similar the two texts are.

[0163] In the specific implementation of edge devices, text can be mapped to the aforementioned vector space using text vectorization tools in Python or C++ environments (such as scikit-learn, OpenVINO, or embedded versions of the Transformer inference library). For the K-means clustering process, the initial number of cluster centers can be specified (e.g., k=5). Then, cosine similarity is measured between the key field vectors of the first text data and multiple second text data vectors. Text with similarity greater than or equal to a certain threshold (e.g., 0.75) is grouped into the same cluster, and this process is iterated until the cluster assignment is stable. If a non-parametric clustering algorithm such as DBSCAN is used, the text data is automatically aggregated into several clusters based on the similarity, and unrelated text is marked as noise or "outliers." Finally, any text belonging to a cluster with high similarity to the first text data is considered to have a high relevance to the current scene, and the system will associate it with the first dataset for subsequent large language model inference.

[0164] Through the above-described exemplary similarity calculation process and common clustering methods, the feasibility of this application in performing text analysis and aggregation on the edge side of the second text data is significantly enhanced. Since algorithms such as cosine similarity and K-means / DBSCAN are already maturely implemented in the existing technical field, those skilled in the art can freely select the most suitable text embedding method (TF-IDF, BERT Embedding, etc.) and clustering algorithm based on hardware conditions and text size to complete the comparison and efficient aggregation with the first text data. This not only facilitates deployment but also achieves a balance between real-time performance and accuracy.

[0165] As an optional implementation, generating the first output result includes:

[0166] The tensor representations and corresponding image and / or audio description text contained in the first dataset are stored in the first partition of the large language model cache to retain key multimodal information;

[0167] The second text data is stored in the second partition of the large language model cache, and indexed or tagged based on the type of the second text data;

[0168] Simultaneously, data from the first and second partitions are read and input into the large language model for inference to obtain the first output result;

[0169] In response to the first output result, some text data in the second partition is updated or replaced, while the tensor representation and corresponding descriptive text in the first partition remain unchanged.

[0170] In this optional implementation, multimodal key information and updatable text data need to be partitioned and managed within the cache of the large language model so that the core context can be preserved during inference and flexible adjustments can be made to new user instructions or log information.

[0171] Specifically, a small Transformer model with a KV cache structure can be deployed on the edge computing device, and the cache can be divided into several segments to store multimodal tensors and corresponding token information.

[0172] After obtaining the first dataset, the multimodal embedding and image / audio description text are converted into key / value tensors during the tokenizer or embedding mapping stage, and these tensors are written to the first partition of the cache, so as to always retain the context of the core information of the scene in subsequent inference.

[0173] Meanwhile, second text data such as user commands and device logs are allocated to the second partition of the cache and indexed according to their type (such as "CMD" or "LOG") so that they can be quickly identified and replaced when needed.

[0174] When inference begins, multimodal key information and updatable user instructions or log text are read synchronously from the first and second partitions, and the input Transformer model is combined to generate the first output result.

[0175] If the first output indicates that certain instructions or logs must be added or deleted, the system will only update the corresponding Key / Value or Text Token index in the second partition, while the multimodal embedding and description text in the first partition will remain unchanged.

[0176] Through this local update and partition management mechanism, the system does not need to repeatedly load or recalculate the core multimodal features during each inference, thus enabling it to maintain fast multi-round inference on resource-constrained devices in home scenarios and respond flexibly to new user commands or environmental changes at any time.

[0177] This approach avoids forgetting key embeddings during long sequence inference and significantly improves the speed and accuracy of adapting to frequent changes in user input and dynamic switching of home scenarios.

[0178] As an optional implementation, updating the second text data based on the first output result, and then interacting with the large language model based on the first dataset and the updated second text data to generate the second output result includes:

[0179] Extract key information indicating replacement, insertion, and deletion operations for the second text data from the first output result;

[0180] In the second partition of the large language model cache, the corresponding second text data is updated according to the key information;

[0181] The tensor representations and image and / or audio description text contained in the first partition remain unchanged;

[0182] The data from the first partition and the updated second partition are re-read and input into the large language model for inference to obtain the second output result.

[0183] In this optional implementation, the operation instructions provided by the first output result need to be further processed so that the system can flexibly update the text data in the second partition before combining it with the first dataset and inputting it into the large language model to generate the final second output result. If the system cannot insert, replace, or delete new instructions or outdated logs, user needs and device log information will not be synchronized with the current multimodal core context in a timely manner, seriously affecting the accuracy and real-time performance of scene reasoning.

[0184] In the specific implementation process, the first output result is first parsed to obtain key information on the replacement, insertion or deletion operations of the second text data.

[0185] For example, a large language model might indicate "remove instruction X", "update log Y", or "insert new configuration Z" in the first output.

[0186] Upon parsing the relevant operation, the system locates the start and end indices of the corresponding text within the second partition of the large language model cache and updates it locally based on key information. Simultaneously, the multimodal embeddings and image and / or audio description text in the first partition remain unchanged to ensure the stability of the core context during long-sequence interactions. Finally, the system merges the updated second partition with the original unchanged first partition and inputs it back into the large language model for inference, thus outputting a new second output result.

[0187] In this way, the system can closely follow the changes in the current situation and user commands after each round of interaction, while still maintaining a high priority on key multimodal information, ultimately achieving more efficient and timely smart home control and decision-making in the resource-constrained environment of the home.

[0188] Based on the same concept, this application also provides an edge computing-based smart home scene control system corresponding to the edge computing-based smart home scene control method. Since the principle of the system in this application is similar to the above-mentioned edge computing-based smart home scene control method, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be described again.

[0189] Reference Figure 3 The diagram shown is a schematic of a smart home scene control system based on edge computing provided in an embodiment of this application. The system includes:

[0190] The acquisition unit 10 is used to acquire multimodal data from the target device and convert the multimodal data into tensor representation through a pre-trained encoder;

[0191] The first processing unit 20 is configured to generate first text data including image and / or audio descriptions based on the tensor representation and prior rules, and combine the tensor representation with the first text data to form a first dataset;

[0192] The second processing unit 30 is used to acquire multiple second text data and perform cluster analysis on the multiple second text data according to the characteristics of the first text data, so as to associate the second text data related to the current home scene with the first dataset; wherein, the second text data includes: user instructions, device logs and scene configuration;

[0193] The generation unit 40 is used to store the first dataset and the second text data in different cache partitions of the large language model deployed in the edge computing device, and interact with the large language model to generate the first output result;

[0194] The adjustment unit 50 is used to update the second text data based on the first output result, and then interact with the large language model based on the first dataset and the updated second text data to generate a second output result; wherein the second output result is used to control smart home devices.

[0195] Those skilled in the art will understand that, in the methods described above in the specific embodiments, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. It should be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0196] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. A smart home scene control method based on edge computing, characterized in that, include: Multimodal data from a target device is acquired, and the multimodal data is converted into a tensor representation using a pre-trained encoder; wherein the target device includes a camera, an audio acquisition device, and an environmental sensor; Based on the tensor representation and prior rules, first text data including image and / or audio descriptions is generated, and the tensor representation is combined with the first text data to form a first dataset. Multiple second text data are acquired, and cluster analysis is performed on the multiple second text data based on the characteristics of the first text data to associate the second text data related to the current home scene with the first dataset; wherein, the second text data includes: user instructions, device logs, and scene configuration; The first dataset and the second text data are stored in different cache partitions of a large language model deployed in an edge computing device, and interact with the large language model to generate a first output result. The second text data is updated based on the first output result, and then the second text data is interacted with the large language model based on the first dataset and the updated second text data to generate a second output result; wherein, the second output result is used to control smart home devices; The multimodal data includes: image data captured by the camera, audio data captured by the audio collector, and temperature, humidity, or infrared information output by the environmental sensor. The step of converting the multimodal data into a tensor representation using a pre-trained encoder includes: extracting features from image data acquired by a camera and audio data acquired by an audio acquisition device based on a lightweight convolutional self-attention hybrid network; vectorizing and mapping the temperature, humidity, or infrared information output by an environmental sensor and fusing it with the features of the image data and audio data; and using a pruning or quantized self-attention module to model the global correlation of the multimodal features generated after the fusion process to obtain the tensor representation corresponding to the multimodal data. The method of using a self-attention module with pruning or quantization to perform global correlation modeling of multimodal features includes: collecting and recording the interaction frequency of user commands within a preset time period; in response to the interaction frequency of the user commands being greater than or equal to a first threshold, setting the quantization precision of the self-attention module to a first bit width; wherein the first bit width is used to control the processing precision of audio and / or image features; and in response to the interaction frequency of the user commands being less than a second threshold, switching the quantization precision of the self-attention module to a preset low bit width.

2. The smart home scene control method based on edge computing according to claim 1, characterized in that, The lightweight convolutional self-attention hybrid network includes: The image data is divided into image blocks, and the feature vectors extracted from the audio data are inserted into the sequence of the image blocks. The image blocks and audio feature vectors are jointly modeled by fusing convolution operations and a multi-head self-attention mechanism.

3. The smart home scene control method based on edge computing according to claim 2, characterized in that, The step of using a self-attention module with pruning or quantization to perform global correlation modeling of multimodal features to obtain the tensor representation corresponding to the multimodal data further includes: Obtain contextual information about the home scene and the status of environmental sensors; Based on the home scene context information and the environmental sensor status, determine the attention heads that are given priority within the target time period, and perform pruning or shutdown operations on the non-priority attention heads in the self-attention module. The attention head that is given priority retains a preset high-precision parameter weight for processing key information that appears in nighttime audio features or daytime image features; The attention head that has undergone the pruning or closing operation and the attention head that receives priority attention jointly complete the global correlation modeling of multimodal features, and output the pruned tensor representation for subsequent inference.

4. The smart home scene control method based on edge computing according to claim 1, characterized in that, The formation of the first dataset includes: Obtain contextual information about the home environment and current user behavior data; Based on the home scene context information and the user's current behavior data, combined with time period characteristics and the status of smart home devices, the content and accuracy of the generated image and audio descriptions are dynamically adjusted. Based on the adjusted descriptive text, key information is extracted from the context using prior rules to generate first text data containing the state of the home scene. The generated first text data is concatenated with the tensor representation to form the first dataset.

5. The smart home scene control method based on edge computing according to claim 4, characterized in that, The step of acquiring multiple second text data and performing cluster analysis on the multiple second text data based on the characteristics of the first text data to associate the second text data related to the current home scene with the first dataset includes: Acquire multiple second text data from user commands, device logs, and scene configurations, and perform word segmentation preprocessing on the second text data; Based on the key fields or contextual information contained in the first text data, a similarity measurement function is constructed; the similarity measurement function is used to measure the degree of correlation between the second text data and the first text data. The similarity metric function is used to perform cluster analysis on the multiple second text data to generate multiple clusters; Clusters of the second text data with a similarity greater than or equal to a preset threshold are associated with the first dataset to form a set of text information that matches the current home scene.

6. The smart home scene control method based on edge computing according to claim 5, characterized in that, The generation of the first output result includes: The tensor representations and corresponding image and / or audio descriptions contained in the first dataset are stored in the first partition of the large language model cache to retain multimodal key information; The second text data is stored in the second partition of the large language model cache, and indexed or tagged based on the type of the second text data; Simultaneously, data from the first and second partitions are read and input into the large language model for inference to obtain the first output result; In response to the first output result, some text data in the second partition is updated or replaced, while the tensor representation and corresponding descriptive text in the first partition remain unchanged.

7. The smart home scene control method based on edge computing according to claim 6, characterized in that, The step of updating the second text data based on the first output result, and then interacting with the large language model based on the first dataset and the updated second text data to generate the second output result includes: Extract key information indicating replacement, insertion, and deletion operations for the second text data from the first output result; In the second partition of the large language model cache, the corresponding second text data is updated according to the key information; The first text data containing tensor representations and image and / or audio descriptions contained in the first partition remains unchanged; The data from the first partition and the updated second partition are re-read and input into the large language model for inference to obtain the second output result.

8. A smart home scene control system based on edge computing, used to implement the smart home scene control method based on edge computing as described in any one of claims 1-7, characterized in that, include: The acquisition unit is used to acquire multimodal data from the target device and convert the multimodal data into tensor representation through a pre-trained encoder; wherein the target device includes a camera, an audio acquisition device, and an environmental sensor; The first processing unit is configured to generate first text data including image and / or audio descriptions based on the tensor representation and prior rules, and combine the tensor representation with the first text data to form a first dataset; The second processing unit is used to acquire multiple second text data and perform cluster analysis on the multiple second text data according to the characteristics of the first text data, so as to associate the second text data related to the current home scene with the first dataset; wherein, the second text data includes: user instructions, device logs and scene configuration; The generation unit is used to store the first dataset and the second text data in different cache partitions of the large language model deployed in the edge computing device, and interact with the large language model to generate the first output result; An adjustment unit is configured to update the second text data based on the first output result, and then interact with the large language model based on the first dataset and the updated second text data to generate a second output result; wherein the second output result is used to control smart home devices; The multimodal data includes: image data captured by the camera, audio data captured by the audio collector, and temperature, humidity, or infrared information output by the environmental sensor. The step of converting the multimodal data into a tensor representation using a pre-trained encoder includes: extracting features from image data acquired by a camera and audio data acquired by an audio acquisition device based on a lightweight convolutional self-attention hybrid network; vectorizing and mapping the temperature, humidity, or infrared information output by an environmental sensor and fusing it with the features of the image data and audio data; and using a pruning or quantized self-attention module to model the global correlation of the multimodal features generated after the fusion process to obtain the tensor representation corresponding to the multimodal data. The method of using a self-attention module with pruning or quantization to perform global correlation modeling of multimodal features includes: collecting and recording the interaction frequency of user commands within a preset time period; in response to the interaction frequency of the user commands being greater than or equal to a first threshold, setting the quantization precision of the self-attention module to a first bit width; wherein the first bit width is used to control the processing precision of audio and / or image features; and in response to the interaction frequency of the user commands being less than a second threshold, switching the quantization precision of the self-attention module to a preset low bit width.

Images