AIoT multimodal data fusion decision-making method, system and application based on deep learning

Abstract

The present invention relates to the field of Internet of Things and deep learning technology, and discloses a deep learning-based AIoT multimodal data fusion decision-making method, system and application, the method comprising acquiring multimodal data for feature extraction, generating feature representations of each modal data; utilizing an adaptive gating weighting mechanism to fuse the feature representations of each modal data, generate a fused feature representation, and convert the fused feature representation into a lightweight feature; uploading the lightweight feature to the cloud, performing high-level feature fusion on the lightweight feature through an adaptive deep learning model pre-deployed in the cloud, and obtaining a high-level fused feature representation; generating a decision result based on the high-level fused feature representation. The present invention improves the capabilities of multimodal data fusion technology in feature extraction, cross-modal correlation mining and real-time processing at the edge, and can meet the needs of intelligent spaces for efficient fusion of multimodal data and low-latency dynamic decision-making.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things and deep learning technology, and more specifically, to a deep learning-based AIoT multimodal data fusion decision-making method, system and application. Background Art

[0002] With the rapid development of Artificial Intelligence for the Internet of Things (AIoT) technology, the construction of smart spaces has become a core research direction in smart cities, smart homes, the Industrial Internet of Things, and other fields. Smart spaces deploy a large number of heterogeneous sensors (such as cameras, temperature sensors, motion detectors, etc.) and smart devices to collect multimodal data (images, audio, environmental parameters, etc.) in real time and provide intelligent services based on this data. However, the heterogeneity of multimodal data, the real-time requirements, and device coordination issues pose significant challenges to existing technologies. Traditional data processing methods struggle to effectively integrate dispersed, multi-source, heterogeneous information, resulting in incomplete environmental perception, high decision-making response latency, and difficulty adapting to the complex demands of dynamic scenarios.

[0003] Currently, existing technologies primarily achieve multimodal data fusion through centralized architectures or layered service models. For example, agricultural IoT solutions based on service grids encapsulate device resources through a unified service framework. However, centralized architectures can easily lead to data processing bottlenecks, making it difficult to meet real-time requirements. Wireless sensor networks based on low-frame-rate cameras reduce deployment costs, but insufficient image resolution and the high complexity of multimodal data fusion algorithms limit their ability to capture dynamic events. Distributed multi-agent systems can share computing pressures, but communication latency and data dependency still hinder decision-making efficiency. These technologies generally suffer from shortcomings in feature extraction, cross-modal correlation mining, and low real-time processing capabilities at the edge, making it difficult to meet the needs of intelligent spaces for efficient multimodal data fusion and low-latency dynamic decision-making. Summary of the Invention

[0004] To overcome the shortcomings of existing multimodal data fusion technologies in feature extraction, cross-modal correlation mining, and real-time edge processing, and to meet the needs of intelligent spaces for efficient fusion of multimodal data and low-latency dynamic decision-making, the present invention proposes the following technical solutions:

[0005] In the first aspect, the present invention proposes an AIoT multimodal data fusion decision-making method based on deep learning, comprising:

[0006] Acquire multimodal data through sensor networks deployed in smart spaces.

[0007] Feature extraction is performed on each modal data to generate feature representation of each modal data.

[0008] The feature representations of each modality data are fused to generate a fused feature representation.

[0009] The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space.

[0010] The lightweight features are uploaded to the cloud, and the high-level feature fusion is performed on the lightweight features through the adaptive deep learning model pre-deployed in the cloud to obtain the high-level fused feature representation.

[0011] Generate decision results based on high-level fusion feature representation.

[0012] As a preferred technical solution, the multimodal data includes image data, audio data and time series data.

[0013] As a preferred technical solution, feature extraction of image data includes:

[0014] For image data I Perform convolution operation Get high-resolution feature maps , medium resolution feature map and low-resolution feature maps .in, and are the weight matrix and bias term for convolution operation respectively.

[0015] Based on the high-resolution feature map , medium resolution feature map and low-resolution feature maps , calculate the query matrix , whose expression is as follows:

[0016]

[0017] Where, 、 and are different linear transformation weight matrices respectively.

[0018] According to the query matrix ,pass H Attention heads are used to calculate the attention output, which is expressed as follows:

[0019]

[0020] The attention outputs of all attention heads are fused to obtain a multi-scale attention fusion feature representation, which is expressed as follows:

[0021]

[0022]

[0023] Where, Represented as a linear transformation weight matrix.

[0024] The direction in which the image data is taken is encoded in position, and its expression is as follows:

[0025]

[0026] Where, The direction of sight for image capture is z The angle between the axes, The rotation angle of the viewing direction of the image in the xy plane.

[0027] The position-encoded direction information enter MLP The model performs feature mapping to obtain directional feature representation, which is expressed as follows:

[0028]

[0029] in, Two-story MLP The weight matrix in the model, is a nonlinear activation function, and Two-story MLP Bias terms in the model.

[0030] The multi-scale attention fusion feature representation and the direction feature representation are fused to obtain the image feature representation , whose expression is as follows:

[0031] .

[0032] As a preferred technical solution, feature extraction of audio data includes:

[0033] The time domain audio signal in the audio data Convert to frequency domain representation , whose expression is as follows:

[0034]

[0035] in, represents the short-time Fourier transform, Corresponding to different time steps t(t )’s frequency signal component, T Indicates the total number of audio clips.

[0036] Representing the frequency domain enter LSTM The model is processed to calculate the audio data at different time steps t The hidden state sequence of the output , whose expression is as follows:

[0037]

[0038] in, represents the long short-term memory network model, for LSTM Model in The cell state at the time step.

[0039] Calculates weighted values based on the energy of the audio data , whose expression is as follows:

[0040]

[0041]

[0042] Where, is the energy of the audio data at time step t, Represents the time step t Mid-frequency signal components The amplitude value, Indicates the total number of frequency signal components.

[0043] According to the hidden state sequence and weighted weights , generates the feature representation of audio data, whose expression is as follows:

[0044] .

[0045] As a preferred technical solution, feature extraction of time series data includes:

[0046] For the original time series data including temperature and humidity , using the second-order autoregressive model for stabilization processing to generate time series data , whose expression is as follows:

[0047]

[0048] in, Indicates the first t The observations at time steps, Indicates the first t The observations at time steps, and Observed values and observations The weight of is the white noise term.

[0049] use LSTM Model and ARIMA Model, for time series data after stationary processing Perform feature extraction to obtain a time series feature set , whose expression is as follows:

[0050]

[0051]

[0052] in, For the extracted t Time series features, represents the autoregressive integrated moving average model, represents the long short-term memory network model, for LSTM Model in The cell state at the time step.

[0053] As a preferred technical solution, an adaptive gating weighting mechanism is used to fuse the feature representations of each modality data to generate a fused feature representation, including:

[0054] The feature representation of each modal data is mapped to a unified feature space, and its expression is as follows:

[0055]

[0056] Where, Indicates the i Unified feature representation of modal data, is the activation function, For the i The projection matrix of the modal data, is the bias term, is the i-th modal data dimension, is the dimension of the final unified feature representation, For the i Feature representation of modal data.

[0057] Constructing a unified feature representation set ,in 、 and They are unified feature representations of image, audio and time series features respectively.

[0058] The adaptive gating weighting mechanism is used to fuse the unified feature representations of each modality data to obtain the initial fused feature representation. , whose expression is as follows:

[0059]

[0060]

[0061] Where, represents the connection operation on the unified feature representation of image, audio and time series features, are the weight matrix and bias term of the adaptive gating weighting mechanism, respectively.

[0062] The initial fusion feature representation is further fused using two fully connected layers to obtain the final fusion feature representation. , whose expression is as follows:

[0063]

[0064] Where, and are the weight matrices of the two fully connected layers respectively, and are the biases of the two fully connected layers respectively.

[0065] As a preferred technical solution, the lightweight deep learning model is a MobileNet model that has undergone pruning and distillation processing.

[0066] The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices, including:

[0067] Use the global average pooling layer in the MobileNet model to represent the fusion features Perform dimensionality reduction and express the fusion features by calculation The average value in the spatial dimension compresses the dimension of the fusion feature to the preset lightweight dimension to generate a lightweight feature .

[0068] As a preferred technical solution, a pre-deployed adaptive deep learning model in the cloud is used to perform high-level feature fusion on lightweight features, including:

[0069] Generate a dynamic convolution kernel according to the following formula :

[0070]

[0071] in, Represents reshaping the generated parameters into the shape of the convolution kernel, is the activation function, and are the weights of the two fully connected layers in the adaptive deep learning model, and are the biases of the two fully connected layers in the adaptive deep learning model, A lightweight feature that has been converted.

[0072] Using dynamically generated convolution kernels , for lightweight features Perform adaptive feature extraction to obtain adaptive features , whose expression is as follows:

[0073]

[0074] Perform feature fusion on the adaptive features to obtain the final high-level feature fusion representation , whose expression is as follows:

[0075]

[0076] in, For the i Adaptive features The weight of N is the number of adaptive features.

[0077] In a second aspect, the present invention further proposes an AIoT multimodal data fusion decision system based on deep learning, which is applied to the AIoT multimodal data fusion decision method based on deep learning as described in any solution of the first aspect, including:

[0078] The acquisition module is used to acquire multimodal data through a sensor network deployed in the smart space. The multimodal data includes image data, audio data and time series data.

[0079] The extraction module is used to extract features from each modal data and generate feature representations of each modal data.

[0080] The first fusion module is used to fuse the feature representations of each modality data to generate a fused feature representation.

[0081] The conversion module is used to convert the fused feature representation into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space.

[0082] The second fusion module is used to upload lightweight features to the cloud, and perform high-level feature fusion on the lightweight features through the adaptive deep learning model pre-deployed in the cloud to obtain high-level fused feature representation.

[0083] The generation module is used to generate decision results based on high-level fusion feature representation.

[0084] In a third aspect, the present invention further proposes an application of the AIoT multimodal data fusion decision method based on deep learning as described in any one of the first aspects in smart space anomaly detection, including:

[0085] Acquire multimodal data through sensor networks deployed in smart spaces.

[0086] Feature extraction is performed on each modal data to generate feature representation of each modal data.

[0087] The feature representations of each modality data are fused to generate a fused feature representation.

[0088] The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space.

[0089] The lightweight features are uploaded to the cloud, and the high-level feature fusion is performed on the lightweight features through the adaptive deep learning model pre-deployed in the cloud to obtain the high-level fused feature representation.

[0090] The high-level fusion feature representation is input into the pre-trained classification model to classify the state of the smart space and identify anomaly detection results.

[0091] The beneficial effects of the present invention include at least:

[0092] (1) The present invention effectively solves the problem of efficient processing of multi-source heterogeneous data in smart spaces by constructing an edge-cloud collaborative multimodal data fusion architecture. Specifically, the sensor network deployed in the smart space collects multimodal data in real time and extracts features from each modal data in combination with a deep learning model, breaking through the traditional method's reliance on single-modal data and improving the comprehensiveness of environmental perception. An adaptive gated weighting mechanism is used to dynamically fuse multimodal features. Through a learnable weight distribution strategy, the ability to mine cross-modal correlations is enhanced, providing a data foundation for accurate decision-making in complex scenarios. The lightweight model deployed on the edge device significantly reduces the amount of cloud data transmission through feature compression and dimensionality reduction processing while ensuring the integrity of key information, reducing network latency and meeting real-time requirements. The cloud-based adaptive deep learning model further strengthens feature expression and generates high-level fusion features. Combined with the multimodal weighted fusion mechanism, the accuracy and robustness of the decision results are improved. Through layered processing and collaborative optimization, this solution builds a closed-loop "perception-processing-decision-making" system in the intelligent space, effectively improving feature extraction efficiency, cross-modal correlation analysis capabilities and real-time response speed at the edge, meeting the core needs of the intelligent space for efficient fusion of multimodal data and low-latency dynamic decision-making.

[0093] (2) The present invention effectively improves the anomaly recognition capability in complex environments by applying a multimodal data fusion decision-making method in intelligent space anomaly detection. Specifically, the deployed multimodal sensor network collects video, audio, and environmental parameter data in real time, and combines it with a deep learning model to extract the features of each modality (such as the multi-scale attention features of video, the temporal pattern of audio, and the dynamic trend of environmental parameters), breaking through the limitations of single-modal detection and enhancing the comprehensiveness and robustness of anomaly features. The adaptive gated weighting mechanism strengthens cross-modal correlation mining by dynamically allocating modal weights, allowing the fusion features to more accurately reflect the comprehensive characteristics of abnormal scenes. The lightweight model of edge devices significantly reduces cloud transmission latency while ensuring the integrity of key information through feature dimensionality reduction and compression, thereby achieving real-time detection of anomaly features. The cloud-based adaptive deep learning model further improves the feature expression capability in complex scenarios through dynamic convolution kernel generation and weighted fusion. Combined with the pre-trained classification model, it achieves efficient classification of multiple types of anomalies such as fire, intrusion, and temperature and humidity exceeding limits, effectively improving the timeliness and accuracy of anomaly detection, and meeting the core requirements of smart spaces for low-latency, high-reliability multimodal data fusion decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 A flowchart of an AIoT multimodal data fusion decision-making method based on deep learning provided by an embodiment of the present invention.

[0095] Figure 2 This is an architectural diagram of the deep learning-based AIoT multimodal data fusion decision-making system provided by an embodiment of the present invention.

[0096] Figure 3 This is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0097] The following will describe embodiments of the present invention with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art will readily understand other advantages and benefits of the present invention from the contents disclosed in this specification. The present invention may also be implemented or applied through different specific embodiments, and the details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0098] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0099] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0100] Example 1

[0101] This embodiment proposes an AIoT multimodal data fusion decision-making method based on deep learning, such as Figure 1 As shown, Figure 1 A flowchart of a deep learning-based AIoT multimodal data fusion decision-making method provided by an embodiment of the present invention includes the following steps:

[0102] S1: Acquire multimodal data through sensor networks deployed in smart spaces.

[0103] In this embodiment, a multimodal sensor network is deployed in a smart space to achieve real-time perception and collection of multi-source data. Specifically, this involves selecting heterogeneous sensors such as cameras, microphones, and temperature and humidity sensors based on application requirements and strategically deploying them to ensure coverage of the target area. A low-latency, high-reliability local sensor network is constructed using wired or wireless methods, enabling efficient inter-device communication based on the CoAP protocol. System clock synchronization ensures temporal consistency of multimodal data, dynamically configuring acquisition frequency and mode to optimize data collection efficiency. Raw data is cleaned and denoised at the sensor end or edge device, removing invalid or redundant information before being temporarily cached on the edge device. This provides a pre-processed data foundation for subsequent feature extraction and fusion. Subsequently, deep learning models (such as convolutional neural networks and long short-term memory networks) are used to extract features from each modality, including image, audio, and environmental parameters, generating semantically rich feature representations that lay the foundation for cross-modal data fusion.

[0104] S2: Extract features from each modal data and generate feature representations of each modal data.

[0105] S3: Using the adaptive gating weighting mechanism, the feature representations of each modality data are fused to generate a fused feature representation.

[0106] S4: The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space.

[0107] S5: Upload the lightweight features to the cloud, and perform high-level feature fusion on the lightweight features through the adaptive deep learning model pre-deployed in the cloud to obtain the high-level fused feature representation.

[0108] S6: Generate decision results based on high-level fusion feature representation.

[0109] In this embodiment, a distributed sensor network is deployed throughout the smart space, responsible for collecting multimodal data in real time. Edge devices pre-deploy lightweight deep learning models to extract and fuse features from each modality of sensor data, converting the fused features into lightweight features. The edge devices then transmit these lightweight features over the network to a remote cloud. The cloud, relying on a pre-deployed adaptive deep learning model, performs high-level fusion of the lightweight features and ultimately generates a decision. This entire process forms a physically coordinated relationship: "sensor acquisition within the smart space—local processing by edge devices within the space—and in-depth analysis by the remote cloud," enabling the full-chain processing of multimodal data from acquisition to decision-making.

[0110] In this embodiment, the decision results of cloud-based analysis are fed back in real time to users or related devices within the smart space, such as smart homes, security equipment, and industrial systems. Through various feedback methods, such as alarms, notifications, and automatic device control, efficient and intelligent responses are achieved, providing users with timely and accurate services and enhancing the user experience of the smart space. The system automatically monitors dynamic changes in environmental data and uses adaptive deep learning techniques and online learning mechanisms to adjust model parameters and optimize feature extraction and fusion strategies, enabling the model to adapt to different environments and changing needs, thereby enhancing the flexibility and accuracy of data processing. Furthermore, a closed-loop feedback system is established, using feedback results and user behavior data as input to improve feature extraction, fusion algorithms, and decision models. Based on historical data and feedback information, the structure and parameters of the deep learning model are gradually optimized, forming a self-optimizing cycle to ensure the long-term stable operation of the system within the smart space.

[0111] It's understandable that building an edge-cloud collaborative multimodal data fusion architecture effectively addresses the challenge of efficiently processing multi-source heterogeneous data in smart spaces. Specifically, sensor networks deployed in smart spaces collect multimodal data in real time and, combined with deep learning models, extract features from each modality. This breaks through the traditional reliance on single-modal data and improves the comprehensiveness of environmental perception. An adaptive gated weighting mechanism dynamically fuses multimodal features. A learnable weight distribution strategy enhances the ability to mine cross-modal correlations, providing a data foundation for accurate decision-making in complex scenarios. The lightweight model deployed on edge devices uses feature compression and dimensionality reduction to significantly reduce the amount of data transmitted to the cloud while ensuring the integrity of critical information, reducing network latency and meeting real-time requirements. The cloud-based adaptive deep learning model further enhances feature representation and generates high-level fused features. Combined with a multimodal weighted fusion mechanism, this improves the accuracy and robustness of decision-making results. Through layered processing and collaborative optimization, this solution builds a closed-loop "perception-processing-decision-making" system in the intelligent space, effectively improving feature extraction efficiency, cross-modal correlation analysis capabilities and real-time response speed at the edge, meeting the core needs of the intelligent space for efficient fusion of multimodal data and low-latency dynamic decision-making.

[0112] Example 2

[0113] This embodiment improves upon the deep learning-based AIoT multimodal data fusion decision-making method proposed in Example 1.

[0114] In this embodiment, the multimodal data includes image data, audio data and time series data.

[0115] In this embodiment, feature extraction of image data includes:

[0116] For image data I Perform convolution operation Get high-resolution feature maps , medium resolution feature map and low-resolution feature maps ;in, and are the weight matrix and bias term for convolution operation respectively; It is a feature map extracted from the high-resolution layer, with higher resolution but weaker semantic information; It is a feature map extracted from the medium-resolution layer, containing balanced local and global information; It is a feature map extracted from a low-resolution layer (deeper layer), which has lower resolution but contains stronger global semantic information.

[0117] Based on the high-resolution feature map , medium resolution feature map and low-resolution feature maps , calculate the query matrix , whose expression is as follows:

[0118]

[0119] Where, 、 and are different linear transformation weight matrices respectively.

[0120] According to the query matrix ,pass H Attention heads are used to calculate the attention output, which is expressed as follows:

[0121]

[0122] The attention outputs of all attention heads are fused to obtain a multi-scale attention fusion feature representation, which is expressed as follows:

[0123]

[0124]

[0125] Where, Represented as a linear transformation weight matrix;

[0126] The direction in which the image data is taken is encoded in position, and its expression is as follows:

[0127]

[0128] Where, The direction of sight for image capture is z The angle between the axes, is the rotation angle of the sight line direction of the image in the xy plane. Performing high-dimensional transformation enables MLP to better learn directional information.

[0129] The position-encoded direction information enter MLP The model performs feature mapping to obtain directional feature representation, which is expressed as follows:

[0130]

[0131] in, Two-story MLP The weight matrix in the model, is a nonlinear activation function, and Two-story MLP Bias terms in the model. It is a perspective fusion representation, which is a feature mapping or encoding method based on MLP, used to enhance the perspective perception ability of the feature extraction network.

[0132] The multi-scale attention fusion feature representation and the direction feature representation are fused to obtain the image feature representation , whose expression is as follows:

[0133] .

[0134] It should be noted that this embodiment builds an enhanced convolutional network EnhancedCNN . Combining convolutional neural networks, attention mechanisms, and multi-scale fusion modules. At the same time, viewpoint and orientation data are added to the image data, such as multi-view images captured by different cameras, and the direction of each image ( ), which facilitates further processing of images required by downstream tasks. EnhancedCNN Process video and image data from cameras , automatically extracting key features of the image through operations such as convolution, multi-scale fusion and attention mechanism , such as object outline, texture, etc.

[0135] In this embodiment, feature extraction of audio data includes:

[0136] The time domain audio signal in the audio data Convert to frequency domain representation , whose expression is as follows:

[0137]

[0138] in, represents the short-time Fourier transform, Corresponding to different time steps t(t )’s frequency signal component, T To indicate the total number of audio clips;

[0139] Representing the frequency domain enter LSTM The model is processed to calculate the audio data at different time steps t The hidden state sequence of the output , whose expression is as follows:

[0140]

[0141] in, represents the long short-term memory network model, for LSTM Model in The cell state at the time step;

[0142] Calculates weighted values based on the energy of the audio data , whose expression is as follows:

[0143]

[0144]

[0145] Where, is the energy of the audio data at time step t, Represents the time step t Mid-frequency signal components The amplitude value, Indicates the total number of frequency signal components;

[0146] According to the hidden state sequence and weighted weights , generates the feature representation of audio data, whose expression is as follows:

[0147] .

[0148] In this embodiment, feature extraction of time series data includes:

[0149] For the original time series data including temperature and humidity , using the second-order autoregressive model for stabilization processing to generate time series data , whose expression is as follows:

[0150]

[0151] in, Indicates the first t The observations at time steps, Indicates the first t The observations at time steps, and Observed values and observations The weight of is the white noise term;

[0152] use LSTM Model and ARIMA Model, for time series data after stationary processing Perform feature extraction to obtain a time series feature set , whose expression is as follows:

[0153]

[0154]

[0155] in, For the extracted t Time series features, represents the autoregressive integrated moving average model, represents the long short-term memory network model, for LSTM Model in The cell state at the time step.

[0156] It should be noted that for time series data such as temperature and humidity, this embodiment uses a long short-term memory network (LSTM) combined with a second-order autoregressive model AR2 to extract its time series features and capture the key patterns of changes in environmental parameters such as temperature and humidity over time. The autoregressive model can observe the linear relationship between the observations at the current time point, thereby improving the local correlation of the sequence. Based on the frequency characteristics of the data, the sliding window method is used to segment the time series and enhance the ability to capture changing trends.

[0157] In this embodiment, the feature representations of each modality data are fused using an adaptive gating weighting mechanism to generate a fused feature representation, including:

[0158] The feature representation of each modal data is mapped to a unified feature space, and its expression is as follows:

[0159]

[0160] Where, Indicates the i Unified feature representation of modal data, is the activation function, For the i The projection matrix of the modal data, is the bias term, is the i-th modal data dimension, is the dimension of the final unified feature representation, For the i Feature representation of modal data;

[0161] Constructing a unified feature representation set ,in 、 and Unified feature representations for image, audio, and temporal features respectively;

[0162] The adaptive gating weighting mechanism is used to fuse the unified feature representations of each modality data to obtain the initial fused feature representation. , whose expression is as follows:

[0163]

[0164]

[0165] Where, represents the connection operation on the unified feature representation of image, audio and time series features, are the weight matrix and bias term of the adaptive gating weighting mechanism respectively;

[0166] The initial fusion feature representation is further fused using two fully connected layers to obtain the final fusion feature representation. , whose expression is as follows:

[0167]

[0168] Where, and are the weight matrices of the two fully connected layers respectively, and are the biases of the two fully connected layers respectively.

[0169] In this embodiment, the lightweight deep learning model is a MobileNet model that has undergone pruning and distillation processing;

[0170] The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices, including:

[0171] Use the global average pooling layer in the MobileNet model to represent the fusion features Perform dimensionality reduction and express the fusion features by calculation The average value in the spatial dimension compresses the dimension of the fusion feature to the preset lightweight dimension to generate a lightweight feature .

[0172] It's important to note that in lightweight processing on edge computing devices, lightweight deep learning models such as MobileNet are selected based on the computing and storage resources of edge devices (such as smart sensor nodes and edge servers). These models are then compressed and optimized using techniques such as L1 pruning, quantization, and distillation. L1 pruning removes redundant neurons and weights, quantization converts floating-point numbers into low-precision representations to reduce model size, and distillation transfers complex model knowledge to improve edge model accuracy. After deployment, a layered processing strategy performs low-latency, low-complexity preliminary analysis and fusion of multimodal data on the edge device, converting raw data into lightweight features or preliminary fused features, significantly reducing the amount of data uploaded to the cloud. Furthermore, the system dynamically adjusts model operating parameters based on real-time task load to ensure edge computing efficiency. The model is regularly updated to adapt to environmental changes. Model parameters are optimized through online learning to maintain the accuracy and adaptability of edge analysis, ultimately achieving an efficient data processing architecture that collaborates between edge computing and the cloud.

[0173] In this embodiment, high-level feature fusion is performed on lightweight features through an adaptive deep learning model pre-deployed in the cloud, including:

[0174] Generate a dynamic convolution kernel according to the following formula :

[0175]

[0176] in, Represents reshaping the generated parameters into the shape of the convolution kernel, is the activation function, and are the weights of the two fully connected layers in the adaptive deep learning model, and are the biases of the two fully connected layers in the adaptive deep learning model, It is a lightweight feature that has been converted;

[0177] Using dynamically generated convolution kernels , for lightweight features Perform adaptive feature extraction to obtain adaptive features , whose expression is as follows:

[0178]

[0179] Perform feature fusion on the adaptive features to obtain the final high-level feature fusion representation , whose expression is as follows:

[0180]

[0181] in, For the i Adaptive features The weight of N is the number of adaptive features.

[0182] It is important to note that the initial analysis and fused data processed by the edge devices is uploaded to the cloud. Leveraging the cloud's high-performance computing resources, a higher level of data fusion is performed. A stable data transmission channel is also established to ensure real-time and secure transmission of edge data, meeting the real-time decision-making requirements of smart spaces. Adaptive convolutional networks are introduced into the cloud-based deep learning model to dynamically adjust feature extraction strategies based on real-time context (such as environmental and temporal changes). This allows the system to adaptively optimize the feature extraction process in different environments or downstream tasks. By analyzing changes in multimodal data, the strategy is automatically updated, improving the accuracy and robustness of data processing. Within a unified feature space on the cloud, deep learning models such as multimodal fusion networks and attention-based networks are used to perform a high-level integration of multimodal features transmitted from the edge. Intelligent fusion strategies such as weighted fusion and feature selection are employed to make the fused data more decision-making-worthy and provide reliable feature support for subsequent intelligent analysis. Ultimately, based on the fused multimodal data, real-time services and responses are provided, improving user experience and overall system efficiency.

[0183] Example 3

[0184] This embodiment proposes an application of the AIoT multimodal data fusion decision-making method based on deep learning as described in the above embodiment in smart space anomaly detection, including:

[0185] Step 1: Acquire multimodal data through the sensor network deployed in the smart space.

[0186] Step 2: Extract features from each modal data and generate feature representations of each modal data.

[0187] Step 3: Fuse the feature representations of each modality data to generate a fused feature representation.

[0188] Step 4: Convert the fused feature representation into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space.

[0189] Step 5: Upload the lightweight features to the cloud, and perform high-level feature fusion on the lightweight features through the adaptive deep learning model pre-deployed in the cloud to obtain the high-level fused feature representation.

[0190] Step 6: Input the high-level fusion feature representation into the pre-trained classification model to classify the state of the smart space and identify the anomaly detection results.

[0191] As an example, in smart campus and smart home scenarios, multimodal sensors such as cameras, temperature and humidity sensors, and CO2 detectors are deployed to collect environmental data in real time. After initial fusion of the modal features by edge devices, the resulting lightweight features are uploaded to the cloud. The cloud uses an adaptive deep learning model to generate high-level fused features, which are then fed into a pretrained classification model (such as a ResNet-based multi-label classifier) to identify anomalies such as fire, intrusion, and temperature and humidity violations. Decision results are pushed to smart terminals in real time, triggering alarms and triggering fire sprinklers or closing doors and windows. Finally, a closed-loop feedback mechanism collects device response data and optimizes the classification model parameters online, significantly improving the security and protection capabilities of smart spaces.

[0192] In the smart city security system, multi-channel camera video streams, microphone ambient sound, and access control card swipe records are subjected to feature extraction and edge fusion, generating lightweight features that are uploaded to the cloud. Dynamic convolution kernels are used in the cloud to enhance feature discrimination. The resulting high-level fused features are then fed into a pre-trained classification model (such as a spatiotemporal Transformer classifier) to accurately identify behaviors such as suspicious individuals wandering around or unusual screaming. Decision results are pushed to security terminals in real time, triggering automatic PTZ camera tracking and generating event reports, achieving an intelligent upgrade from passive monitoring to active early warning.

[0193] It's understandable that applying multimodal data fusion decision-making methods to smart space anomaly detection effectively improves anomaly identification capabilities in complex environments. Specifically, the deployed multimodal sensor network collects video, audio, and environmental parameter data in real time. Combined with deep learning models, it specifically extracts features from each modality (such as multi-scale attention features in video, temporal patterns in audio, and dynamic trends in environmental parameters). This overcomes the limitations of single-modal detection and enhances the comprehensiveness and robustness of anomaly features. The adaptive gated weighting mechanism dynamically assigns modal weights, enhancing cross-modal correlation mining and enabling fused features to more accurately reflect the comprehensive characteristics of anomaly scenarios. The lightweight edge device model significantly reduces cloud transmission latency while ensuring the integrity of critical information through feature dimensionality reduction and compression, enabling real-time detection of anomaly features. The cloud-based adaptive deep learning model further improves the feature expression capability in complex scenarios through dynamic convolution kernel generation and weighted fusion. Combined with the pre-trained classification model, it achieves efficient classification of multiple types of anomalies such as fire, intrusion, and temperature and humidity exceeding limits, effectively improving the timeliness and accuracy of anomaly detection, and meeting the core requirements of smart spaces for low-latency, high-reliability multimodal data fusion decision-making.

[0194] It should be noted that the above explanation of the embodiment of the AIoT multimodal data fusion decision-making method based on deep learning is also applicable to the application of this embodiment in smart space anomaly detection and will not be repeated here.

[0195] Example 4

[0196] like Figure 2 As shown, this embodiment proposes an AIoT multimodal data fusion decision system based on deep learning, which is applied to the AIoT multimodal data fusion decision method based on deep learning as described in the above embodiment, including: an acquisition module 100, an extraction module 200, a first fusion module 300, a conversion module 400, a second fusion module 500 and a generation module.

[0197] The acquisition module 100 is used to acquire multimodal data from a sensor network deployed in the smart space. The multimodal data includes image data, audio data, and time series data. The extraction module 200 is used to extract features from each modal data and generate feature representations for each modal data. The first fusion module 300 is used to fuse the feature representations of each modal data to generate a fused feature representation. The conversion module 400 is used to convert the fused feature representation into lightweight features using a lightweight deep learning model pre-deployed on edge devices in the smart space. The second fusion module 500 is used to upload the lightweight features to the cloud and perform high-level feature fusion on the lightweight features using an adaptive deep learning model pre-deployed on the cloud to obtain a high-level fused feature representation. The generation module 600 is used to generate a decision result based on the high-level fused feature representation.

[0198] It should be noted that the above explanation of the embodiment of the AIoT multimodal data fusion decision-making method based on deep learning is also applicable to the AIoT multimodal data fusion decision-making system based on deep learning in this embodiment, and will not be repeated here.

[0199] Example 5

[0200] Figure 3 This is a schematic diagram of the structure of an electronic device 700 provided in this embodiment. The electronic device 700 includes: a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor 702.

[0201] When the processor 702 executes the program, it implements the AIoT multimodal data fusion decision method based on deep learning provided in the above embodiment.

[0202] Furthermore, the electronic device 700 further includes a communication interface 703 for communication between the memory 701 and the processor 702 .

[0203] The memory 701 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0204] If the memory 701, the processor 702, and the communication interface 703 are implemented independently, the communication interface 703, the memory 701, and the processor 702 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0205] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0206] The processor 702 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0207] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0208] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0209] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0210] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0211] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above-mentioned embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0212] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A deep learning-based AIoT multimodal data fusion decision-making method, characterized by: include: Acquiring multimodal data through a sensor network deployed in a smart space; the multimodal data includes image data, audio data, and time series data; Extract features from each modal data to generate feature representations of each modal data; Using the adaptive gating weighting mechanism, the feature representations of each modality data are fused to generate a fused feature representation, including: The feature representation of each modal data is mapped to a unified feature space, and its expression is as follows: Where, Indicates the i Unified feature representation of modal data, is the activation function, For the i The projection matrix of the modal data, is the bias term, is the i-th modal data dimension, is the dimension of the final unified feature representation, For the i Feature representation of modal data; Constructing a unified feature representation set ,in 、 and Unified feature representations for image, audio, and temporal features respectively; The adaptive gating weighting mechanism is used to fuse the unified feature representations of each modality data to obtain the initial fused feature representation. , whose expression is as follows: Where, represents the connection operation on the unified feature representation of image, audio and time series features, are the weight matrix and bias term of the adaptive gating weighting mechanism respectively; The initial fusion feature representation is further fused using two fully connected layers to obtain the final fusion feature representation. , whose expression is as follows: Where, and are the weight matrices of the two fully connected layers respectively, and are the biases of the two fully connected layers respectively; The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space; Upload lightweight features to the cloud, and use the adaptive deep learning model pre-deployed in the cloud to perform high-level feature fusion on the lightweight features to obtain the high-level fused feature representation; Generate decision results based on high-level fusion feature representation.

2. The AIoT multimodal data fusion decision-making method based on deep learning according to claim 1 is characterized in that: Perform feature extraction on image data, including: For image data I Perform convolution operation Get high-resolution feature maps , medium resolution feature map and low-resolution feature maps ;in, and are the weight matrix and bias term for convolution operation respectively; Based on the high-resolution feature map , medium resolution feature map and low-resolution feature maps , calculate the query matrix , whose expression is as follows: Where, 、 and are different linear transformation weight matrices respectively; According to the query matrix ,pass H Attention heads are used to calculate the attention output, which is expressed as follows: The attention outputs of all attention heads are fused to obtain a multi-scale attention fusion feature representation, which is expressed as follows: Where, Represented as a linear transformation weight matrix; The direction in which the image data is taken is encoded in position, and its expression is as follows: Where, The direction of sight for image capture is z The angle between the axes, The rotation angle of the sight line direction of the image capture on the xy plane; The position-encoded direction information enter MLP The model performs feature mapping to obtain directional feature representation, which is expressed as follows: in, Two-story MLP The weight matrix in the model, and is a nonlinear activation function, and Two-story MLP Bias terms in the model; The multi-scale attention fusion feature representation and the direction feature representation are fused to obtain the image feature representation , whose expression is as follows: 。 3. The AIoT multimodal data fusion decision-making method based on deep learning according to claim 1 is characterized in that: Perform feature extraction on audio data, including: The time domain audio signal in the audio data Convert to frequency domain representation , whose expression is as follows: in, represents the short-time Fourier transform, Corresponding to different time steps t(t )’s frequency signal component, T To indicate the total number of audio clips; Representing the frequency domain enter LSTM The model is processed to calculate the audio data at different time steps t The hidden state sequence of the output , whose expression is as follows: in, represents the long short-term memory network model, for LSTM Model in The cell state at the time step; Calculates weighted values based on the energy of the audio data , whose expression is as follows: Where, is the energy of the audio data at time step t, Represents the time step t Mid-frequency signal components The amplitude value, Indicates the total number of frequency signal components; According to the hidden state sequence and weighted weights , generates the feature representation of audio data, whose expression is as follows: 。 4. The AIoT multimodal data fusion decision-making method based on deep learning according to claim 1 is characterized in that: Perform feature extraction on time series data, including: For the original time series data including temperature and humidity , using the second-order autoregressive model for stabilization processing to generate time series data , whose expression is as follows: in, Indicates the first t The observations at each time step, Indicates the first t The observations at each time step, and Observed values and observations The weight of is the white noise term; use LSTM Model and ARIMA Model, for time series data after stationary processing Perform feature extraction to obtain a time series feature set , whose expression is as follows: in, For the extracted t Time series features, represents the autoregressive integrated moving average model, represents the long short-term memory network model, for LSTM Model in The cell state at the time step.

5. The AIoT multimodal data fusion decision-making method based on deep learning according to claim 1 is characterized in that: The lightweight deep learning model is a MobileNet model that has undergone pruning and distillation processing; The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices, including: Use the global average pooling layer in the MobileNet model to represent the fusion features Perform dimensionality reduction and express the fusion features by calculation The average value in the spatial dimension compresses the dimension of the fusion feature to the preset lightweight dimension to generate a lightweight feature .

6. The AIoT multimodal data fusion decision-making method based on deep learning according to claim 5 is characterized in that: High-level feature fusion of lightweight features using an adaptive deep learning model pre-deployed in the cloud, including: Generate a dynamic convolution kernel according to the following formula : in, Represents reshaping the generated parameters into the shape of the convolution kernel, is the activation function, and are the weights of the two fully connected layers in the adaptive deep learning model, and are the biases of the two fully connected layers in the adaptive deep learning model, It is a lightweight feature that has been converted; Using dynamically generated convolution kernels , for lightweight features Perform adaptive feature extraction to obtain adaptive features , whose expression is as follows: Perform feature fusion on the adaptive features to obtain the final high-level feature fusion representation , whose expression is as follows: in, For the i Adaptive features The weight of N is the number of adaptive features.

7. A deep learning-based AIoT multimodal data fusion decision system, applied to the deep learning-based AIoT multimodal data fusion decision method according to any one of claims 1 to 6, characterized in that: include: An acquisition module, configured to acquire multimodal data through a sensor network deployed in the smart space; the multimodal data includes image data, audio data, and time series data; The extraction module is used to extract features from each modal data and generate feature representations of each modal data; The first fusion module is used to fuse the feature representations of each modality data to generate a fused feature representation; A conversion module, which is used to convert the fused feature representation into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space; The second fusion module is used to upload lightweight features to the cloud and perform high-level feature fusion on the lightweight features through the adaptive deep learning model pre-deployed in the cloud to obtain the high-level fused feature representation; The generation module is used to generate decision results based on high-level fusion feature representation.

8. An application of the AIoT multimodal data fusion decision-making method based on deep learning in intelligent space anomaly detection according to any one of claims 1 to 6, characterized in that: include: Acquire multimodal data through sensor networks deployed in smart spaces; Extract features from each modal data to generate feature representations of each modal data; Fusing the feature representations of each modality data to generate a fused feature representation; The fused feature representation is converted into lightweight features through a lightweight deep learning model pre-deployed on edge devices in the smart space; Upload lightweight features to the cloud, and use the adaptive deep learning model pre-deployed in the cloud to perform high-level feature fusion on the lightweight features to obtain the high-level fused feature representation; The high-level fusion feature representation is input into the pre-trained classification model to classify the state of the smart space and identify the anomaly detection results.

Images