Lightweight multi-scale adaptive multi-modal data fusion method and system

Abstract

The invention discloses a lightweight multi-scale adaptive multi-modal data fusion method and system, and aims to solve the problem of insufficient robustness caused by high calculation complexity and fusion strategy solidification during deployment on edge equipment in the prior art. The method comprises the following steps: preprocessing multi-modal data; extracting features of each mode in different receptive fields through a lightweight parallel multi-scale network; adaptively generating a modal fusion weight based on a dynamic gating mechanism in each scale; and aggregating the multi-scale fusion features and outputting a classification result. The system comprises a data preprocessing module, a multi-scale feature extraction module, a dynamic gating fusion module, a cross-scale aggregation module and a classification module. According to the method, through a gating fusion mechanism of depth separable convolution and data driving, while the calculation overhead is remarkably reduced, suppression of a noise mode and enhancement of an information enrichment mode are realized, and the real-time performance, robustness and accuracy of the model in a complex environment are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and in particular, relates to a lightweight multi-scale adaptive multi-modal data fusion method and system. BACKGROUND

[0002] With the in-depth application of artificial intelligence technology in edge computing scenarios such as intelligent terminals, autonomous driving, and industrial Internet of Things, multi-modal data fusion has become a key means to improve perception and decision-making capabilities. Multi-modal learning integrates information from various heterogeneous data sources such as text, speech, images, and sensors, enabling a more comprehensive description of complex scene semantics and significantly improving the model's expression ability and generalization performance. However, such methods usually rely on deep neural network architectures with high computational complexity, which poses a serious challenge when deployed on resource-constrained edge devices. Edge devices are generally limited by computing power, memory, and energy consumption, making it difficult to support the large parameter size and intensive computational requirements of traditional multi-modal models, resulting in high inference delay, low energy efficiency, and even inability to run in real time.

[0003] Among them, lightweight multi-modal fusion technology aims to achieve efficient and robust cross-modal information integration under limited resources. The core goal of this direction is to compress the model size while preserving the key semantic features of each modality and dynamically coordinating the contributions of different modalities through adaptive mechanisms. However, existing lightweight methods mostly adopt static fusion strategies such as simple concatenation, weighted averaging, or fixed gating, which cannot be dynamically adjusted according to input content characteristics or device running state. Such methods are prone to sudden performance degradation when facing modality quality differences (such as low-light images, noisy speech) or partial modality absence. In addition, existing solutions often ignore the collaborative use of multi-scale information, focusing only on global semantics while ignoring local details, or vice versa, making it difficult to balance computational overhead and representation ability.

[0004] In existing technologies, although some research attempts to introduce attention mechanisms or gating networks to enhance the flexibility of fusion, their structures are complex, with redundant parameters, and lack the ability to perceive the dynamic computing power environment of edge devices. At the same time, most models do not explicitly model the modality absence scenario during the training phase, resulting in a lack of robustness to real-world problems such as sensor failure and communication interruption. In typical edge multi-modal tasks such as emotion recognition, human-computer interaction, and intelligent monitoring, the above defects are particularly prominent: models either lack sufficient precision due to excessive simplification or cannot meet real-time requirements due to computational overload. Therefore, there is an urgent need for a lightweight multi-modal data fusion method that can balance multi-scale feature expression and adaptively adjust the fusion strategy according to content and device state, and has strong modality absence robustness, to break through the performance and efficiency bottleneck of edge intelligent applications. SUMMARY

[0005] The purpose of the present application is to provide a lightweight multi-scale adaptive multi-modal data fusion method and system, which mainly solves the problem that the existing technology faces high model calculation complexity, high power consumption and difficulty in real-time processing when deploying a multi-modal data fusion model on a resource-limited edge computing device, and the fixed fusion strategy cannot dynamically adjust the contribution of each modality according to the real-time quality of the input data, resulting in a decline in fusion performance and insufficient robustness in complex environments.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows: A lightweight multi-scale adaptive multi-modal data fusion method, comprising the following steps: S1, obtaining a plurality of heterogeneous original modality data streams; S2, performing data preprocessing on the original modality data stream to generate aligned multi-modal data segments with a fixed time length; S3, inputting the multi-modal data segments into a lightweight parallel multi-scale feature extraction network to obtain a set of multi-modal multi-scale feature vectors; S4, inputting the multi-modal multi-scale feature vector set into a scale-in dynamic gated fusion unit to generate a fusion feature vector at this scale; S5, performing cross-scale feature aggregation on the fusion feature vectors generated at all scales to generate a global fusion feature descriptor; S6, inputting the global fusion feature descriptor into a classifier, and the classifier calculates through a sequence linear layer and an activation function to output the final classification result of the multi-modal data segment.

[0007] Further, in the step S1, the original modality data stream includes at least two different data types, and the data types are selected from video data stream, audio data stream, inertial measurement unit data stream or radar point cloud data stream.

[0008] Further, in the step S3, the multi-scale feature extraction network sets an independent feature extraction channel for each modality data, and each feature extraction channel includes at least two parallel feature extraction branches with different receptive field scales, which use depth separable convolution operation to synchronously extract deep feature maps of each modality at different spatio-temporal granularities. The feature extraction channel for the video data stream has parallel feature extraction branches including a small scale branch, a medium scale branch and a large scale branch; the small scale branch uses a 3x3 convolution kernel with a step of 1; the medium scale branch uses a 5x5 convolution kernel with a step of 2; the large scale branch uses a 7x7 convolution kernel with a step of 2 after setting a max pooling layer at the input end; all convolution operations are depth separable convolution, which is composed of a depth convolution and a point convolution in series, and a batch normalization layer and a nonlinear activation function layer are configured after the convolution layer; the feature maps output by each branch generate video feature vectors of corresponding scales after global average pooling; The feature extraction channel for the audio data stream, the inertial measurement unit data stream or the radar point cloud data stream uses multi-scale depth separable convolution branches symmetrical to the video channel structure for feature extraction, and outputs feature vectors of corresponding modalities at each scale; Finally, the feature vectors of all modalities at all scales are combined into the multi-modal multi-scale feature vector set.

[0009] Further, the specific process of the step S4 is as follows: S41, for any scale, the feature vectors of all modalities at the scale are input one by one into a gating weight generation network, and the gating weight generation network is a two-layer multilayer perceptron; S42, the first layer of the two-layer multilayer perceptron maps the input feature vectors to a low-dimensional hidden space, and the second layer maps the low-dimensional hidden space representation to a scalar value; S43, the contribution score of all modalities is normalized by using a soft maximum function to generate a set of dynamic fusion weights with a sum of 1; S44, the feature vectors of the corresponding modalities are weighted and summed using the dynamic fusion weights to generate the fusion feature vector at the scale.

[0010] Further, the gating weight generation network adopts a shared parameter structure when processing feature fusion at different scales, and forces the network to learn a value evaluation function based on the feature content itself regardless of the modality.

[0011] The specific process of the step S5 is as follows: S51, the fusion feature vectors at each scale are spliced along the feature dimension to form a high-dimensional spliced vector; S52, the high-dimensional spliced vector is input into a fully connected layer, and the fully connected layer maps the input dimension to a preset fixed dimension, and outputs the global fusion feature descriptor.

[0012] Further, the specific process of the step S6 is as follows: S61, mapping the global fusion feature descriptor to a dimension of the total number of classification categories through a linear layer; S62, applying a soft-max function to the mapping result to generate a probability vector, each element of the probability vector representing the probability of the input data belonging to the corresponding category; S63, selecting the category corresponding to the maximum value in the probability vector as the final classification result.

[0013] Further, the data preprocessing includes timestamp alignment, data frame segmentation and normalization operation; wherein the timestamp alignment is realized by a sliding window interpolation method, and the timestamps of video frames are used as the basis to perform linear interpolation or nearest neighbor interpolation on other modal data, so that all modal data are aligned to the same time grid in the time dimension; the data frame segmentation adopts a fixed time window mechanism; the normalization operation is performed respectively for each modality, and the numerical value is mapped to the interval of 0 to 1 or range normalization is performed.

[0014] The application also provides a lightweight multi-scale adaptive multi-modal data fusion system for realizing the fusion method, comprising: A multi-modal data acquisition and preprocessing module is configured to capture raw modal data streams through a plurality of heterogeneous sensor interfaces, and perform timestamp alignment, data frame segmentation based on a fixed time window and numerical normalization, and output standardized multi-modal data segments. A lightweight parallel multi-scale feature extraction module is configured to have independent feature extraction channels for each modality, and each channel includes a plurality of parallel feature extraction branches. A scale-in dynamic gating fusion module includes a gating weight generation network. A cross-scale feature aggregation module is configured to receive a group of fusion feature vectors, integrate multi-scale information into a single, fixed-dimension global fusion feature descriptor through vector splicing and fully connected network mapping. A task classification module is configured to receive the global fusion feature descriptor, calculate through a feedforward neural network classifier, and output a final classification decision result.

[0015] Further, in the application, the gating weight generation network is a shared parameter double-layer multilayer perceptron for uniformly evaluating the contribution of each modality feature at each scale.

[0016] Compared with the prior art, the application has the following beneficial effects: (1) The application replaces the traditional convolution operation with depth separable convolution, and designs independent lightweight feature extraction channels for each modality, greatly reducing the model parameter quantity and calculation quantity. For example, the multi-scale branch of the video data stream adopts 3x3, 5x5, 7x7 depth separable convolution, while maintaining the feature extraction capability, effectively reducing the computational complexity (compared with the same performance traditional convolution network). The lightweight design enables the model to run in real time on edge devices with limited computing power (such as embedded terminals, intelligent sensors), solving the deployment problem caused by the overload of traditional multi-modal fusion model.

[0017] (2) The application proposes a scale-in dynamic gating fusion unit: through a double-layer multi-layer perception machine sharing parameters to evaluate the contribution of each modality feature, combined with a soft maximum function to generate dynamic weights, to achieve adaptive suppression of noisy modalities (such as low-light videos, noisy audio) and enhancement of information-rich modalities (such as clear radar point clouds, stable inertial measurement data). Experiments show that in a scene where 30% of the modality data is disturbed by noise, the model classification accuracy remains above 85%, an increase of 12% over the static fusion strategy.

[0018] (3) The application synchronously captures deep features of each modality at different temporal and spatial scales (such as local texture and global scene of video, short-time spectrum and long-time trend of audio), and generates global fusion feature descriptors through cross-scale aggregation. This mechanism solves the limitation of traditional models that only focus on single-scale information, enabling the model to more comprehensively describe complex scene semantics. In the multi-modal emotion recognition task, the classification accuracy is effectively improved compared with the single-scale fusion method.

[0019] (4) In the preprocessing stage, the sliding window interpolation method is used to realize the alignment of multi-modal timestamps (based on video frames, with an accuracy of μs level), combined with fixed time window segmentation and normalization operation, to ensure the spatio-temporal consistency of input data. At the same time, data augmentation strategies (such as video random cropping, audio noise injection) are introduced in the training process to enhance the model's adaptability to dynamic disturbances in edge environments such as sensor failure and communication delay. In actual deployment, the model's fault tolerance rate for modality loss (such as single modality data interruption) is improved, meeting the needs of real-time decision-making scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The figure is a flowchart of the method of the application.

[0021] Figure 2 The figure is a schematic diagram of the core principle framework of the scale-in dynamic gating fusion mechanism in the application.

[0022] Figure 3 The figure is a system architecture diagram of the application. DETAILED DESCRIPTION

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0024] like Figure 1 , 2 As shown, this invention discloses a lightweight multi-scale adaptive multimodal data fusion method and system. The method includes the following steps: acquiring multiple heterogeneous original modal data streams; preprocessing the original modal data streams; inputting the preprocessed multimodal data segments into a lightweight parallel multi-scale feature extraction network; inputting the extracted multimodal multi-scale feature vector set into a scale-intra-scale dynamic gating fusion unit; performing cross-scale feature aggregation on the fused feature vectors at each scale; and inputting the generated global fusion feature descriptor into a classifier to output the final classification result.

[0025] In this embodiment, the original modal data stream contains at least two different types of data, selected from video data stream, audio data stream, inertial measurement unit (IMU) data stream, or radar point cloud data stream. The video data stream is acquired via an image sensor array at a fixed frame rate of 30 frames per second, with each frame having a resolution of 640×480 pixels and three color channels (red, green, and blue). The audio data stream is acquired via a microphone array at a sampling rate of 16kHz and a sampling bit depth of 16 bits. The IMU data stream is jointly output by a three-axis accelerometer and a three-axis gyroscope, with a sampling frequency of 100Hz, and the output data includes linear acceleration and angular velocity. The radar point cloud data stream is generated by a millimeter-wave radar module, outputting 10 frames of point cloud data per second, with each frame containing no more than 500 three-dimensional spatial points and their reflection intensity information. All modal data streams are synchronously input to the system main control unit through independent hardware interfaces, accompanied by high-precision timestamps generated by a unified system clock source with an accuracy of at least μs.

[0026] The raw modal data stream is then preprocessed. Preprocessing includes timestamp alignment, data frame segmentation, and normalization to generate aligned, fixed-duration multimodal data segments. Timestamp alignment is achieved using sliding window interpolation: based on the timestamps of video frames, linear or nearest-neighbor interpolation is performed on audio, inertial measurement unit (IMU), and radar point cloud data, ensuring all modal data are strictly aligned to the same time grid with a grid interval of 33ms, corresponding to the video frame rate. Data frame segmentation employs a fixed time window mechanism with a window length of 2 seconds and a sliding step of 0.5 seconds, thus dividing the continuous data stream into a series of overlapping 2-second data segments. Normalization is performed separately for each modality: pixel values ​​in video data are linearly mapped to the 0-1 range; waveform amplitudes in audio data are divided by the maximum absolute value within the historical sliding window; acceleration and angular velocity in IMU data are divided by the upper limit of their sensor range; and the spatial coordinates of the radar point cloud are centered and scaled to a unit sphere. After the above processing, standardized multimodal data segments are output. Each segment contains a video frame sequence, an audio sampling sequence, an inertial measurement sequence, and a radar point cloud sequence aligned within 2 seconds.

[0027] Next, the multimodal data segments are input into a lightweight parallel multi-scale feature extraction network. This network sets up an independent feature extraction channel for each modality, and each channel contains at least two parallel feature extraction branches with different receptive field scales. All branches use depthwise separable convolution operations to simultaneously extract deep feature maps of each modality at different spatiotemporal granularities, and finally output a set of multimodal multi-scale feature vectors.

[0028] For feature extraction from video data streams, three parallel branches are included: a small-scale branch, a medium-scale branch, and a large-scale branch. The small-scale branch uses a 3×3 convolutional kernel with a stride of 1 to extract fine texture and edge features. The medium-scale branch uses a 5×5 convolutional kernel with a stride of 2 to extract target shape features in local regions. The large-scale branch first passes the input through a max-pooling layer (kernel size 2×2, stride 2), followed by a 7×7 convolutional kernel with a stride of 2, to extract macroscopic scene and contextual features. All convolutional operations are depthwise separable convolutions, consisting of one depthwise convolutional layer and one pointwise convolutional layer cascaded together. The depthwise convolutional layer performs independent spatial convolution on each channel of the input feature map, while the pointwise convolutional layer linearly combines the channels output by the depthwise convolution using a 1×1 convolutional kernel. Each convolutional layer is followed by a batch normalization layer and a modified linear unit activation function. The output feature maps of the three branches are compressed into feature vectors with a spatial dimension of 1 through global average pooling, with dimensions of 128, 256 and 512 respectively, forming the feature vectors of the video modality at the three scales.

[0029] For the feature extraction channels of the audio data stream, three parallel branches are also set up, corresponding to short-time, medium-time, and long-time spectral features, respectively. Each branch first converts the audio sequence into a Mel spectrogram, and then processes it using depthwise separable convolutional kernels of different scales: the small-scale branch uses a 3×3 convolutional kernel with a stride of 1; the medium-scale branch uses a 5×5 convolutional kernel with a stride of 2; and the large-scale branch first performs max pooling and then uses a 7×7 convolutional kernel with a stride of 2. The output feature vector dimension is consistent with that of the video channels.

[0030] The processing channel structure of the inertial measurement unit (IMU) data stream is similar to that of the radar point cloud data stream; both utilize one-dimensional or two-dimensional depthwise separable convolutional networks to extract multi-scale features. Features of all modes at each scale are analyzed. The vectors are output uniformly, forming a set of multimodal, multi-scale feature vectors, denoted as . , where s∈{1,2,3} represents the scale index, and v, a, i, and r represent video, audio, inertial measurement, and radar modes, respectively.

[0031] The multimodal, multi-scale feature vector set is then input into a single-scale dynamic gating fusion unit. This fusion unit performs the fusion operation independently for each scale *s*. For scale *s*, it first processes the feature vectors of all modalities at that scale. (m∈{v,a,i,r}) are sequentially input into a gated weight generation network. This network is a two-layer multilayer perceptron. The first layer maps the input feature vector to a low-dimensional latent space of sixty-four dimensions, and the second layer maps the latent space representation to a scalar value, which is the contribution score of mode m at scale s. The contribution scores of all modalities constitute a vector. Subsequently, regarding Normalization is performed using a soft maximum function to generate dynamic fusion weights. ,in: In the formula, exp represents an exponential function. Finally, the feature vectors of the corresponding modalities are weighted and summed using dynamic fusion weights to generate the fused feature vector at that scale. : This process is performed in parallel across three scales, outputting three fused feature vectors. , , .

[0032] Next, cross-scale feature aggregation is performed on the fused feature vectors generated at all scales. The aggregation operation first... , , The vectors are concatenated along the feature dimension to form a high-dimensional concatenated vector. F concat Its dimensions are 128 + 256 + 512, totaling 896 dimensions. Then, F concat The input is fed into a fully connected layer, which maps the input dimension to 512 dimensions and outputs a globally fused feature descriptor. F global The fully connected layer is followed by a discard layer with a preset discard rate of 0.5. During the model training phase, 50% of the neurons' outputs are randomly set to 0 to prevent overfitting and enhance the model's generalization ability.

[0033] Finally, the globally fused feature descriptor is input into a classifier. This classifier consists of two linear layers and an activation function. The first linear layer maps the 512-dimensional input to the number of classes C, the second linear layer is an identity mapping, and the activation function is a soft-maximum function. Its output is a C-dimensional probability vector P = [p1, p2, ..., p C ], where p c This represents the probability that the input multimodal data segment belongs to class c. The final classification result is the class index corresponding to the maximum probability.

[0034] Throughout the methodology, model training is achieved using the backpropagation algorithm. The loss function employed is cross-entropy loss, defined as: in y c One-hot encoding of the true labels was used; the optimizer employed an adaptive moment estimation optimizer with an initial learning rate of 0.001, momentum parameters β1 of 0.9, and β2 of 0.999. Data augmentation strategies were introduced during training: random cropping and random horizontal flipping were applied to video frames; Gaussian white noise was injected into the audio signal with a signal-to-noise ratio of at least 20 dB; random offset perturbations were applied to the inertial measurement data with an amplitude not exceeding 5% of the measurement range; and random point deletion was performed on the radar point cloud with a deletion ratio not exceeding 10%. All augmentation operations were completed during the data preprocessing stage.

[0035] like Figure 3 As shown, this embodiment also provides a lightweight multi-scale adaptive multimodal data fusion system, including a multimodal data acquisition and preprocessing module, a lightweight parallel multi-scale feature extraction module, an intra-scale dynamic gating fusion module, a cross-scale feature aggregation module, and a task classification module.

[0036] The multimodal data acquisition and preprocessing module captures raw modal data streams through multiple heterogeneous sensor interfaces and performs timestamp alignment, data frame segmentation based on fixed time windows, and numerical normalization to output standardized multimodal data fragments. This module incorporates a high-precision clock synchronization unit to ensure time consistency across sensor data.

[0037] The lightweight parallel multi-scale feature extraction module internally configures an independent feature extraction channel for each modality, and each channel contains multiple parallel feature extraction branches. Each branch forms a different receptive field scale by configuring convolutional kernels of different sizes (3×3, 5×5, 7×7) and different timing parameters (1 or 2). All convolutional operations adopt a depthwise separable convolutional structure, which consists of depthwise convolutional layers and pointwise convolutional layers in series, significantly reducing the amount of computation and parameters.

[0038] The scale-gated fusion module includes a gating weight generation network, which is a two-layer multilayer perceptron with shared parameters. When processing feature fusion at any scale, all modality feature vectors at that scale have their contribution scores calculated through this shared network. The sharing mechanism forces the network to learn a modality-independent value evaluation function based on the feature content itself, thereby reducing model complexity while ensuring adaptability.

[0039] The cross-scale feature aggregation module receives fused feature vectors from various scales and integrates multi-scale information into a single global fused feature descriptor through vector concatenation and mapping with a fully connected network. The fully connected layers in this module are followed by a discard layer with a discard rate of 0.5 to improve model robustness.

[0040] The task classification module receives the globally fused feature descriptor and calculates the final classification decision result through a feedforward neural network classifier. The classifier output layer uses a soft maximum activation function to generate the class probability distribution.

[0041] The system also includes a model training module that uses the cross-entropy loss function, updates parameters through backpropagation and an adaptive moment estimation optimizer, and integrates a data augmentation mechanism to improve generalization ability. All modules can be embedded in a field-programmable gate array or an application-specific integrated circuit to form a hardware accelerator for edge computing devices, enabling low-power, high-energy-efficiency multimodal data fusion processing.

[0042] This embodiment achieves efficient and robust multimodal data fusion on edge devices through the aforementioned methods and system architecture. Depthwise separable convolution and parallel multi-scale architecture significantly reduce computational burden; intra-scale dynamic gating mechanism enables real-time adaptive adjustment of modality weights; and cross-scale aggregation comprehensively integrates multi-level feature information. The entire solution ensures classification accuracy while meeting the stringent requirements of edge computing for low latency and low power consumption.

[0043] The above embodiments are merely one of the preferred embodiments of the present invention and should not be used to limit the scope of protection of the present invention. Any modifications or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but solve the same technical problem as the present invention, should be included within the scope of protection of the present invention.

Claims

1. A lightweight multi-scale adaptive multimodal data fusion method, characterized in that, Includes the following steps: S1, acquire the raw modal data of the multi-path heterogeneous system; S2, perform data preprocessing on the original modal data stream to generate aligned, fixed-duration multimodal data segments; S3, input the multimodal data segment into a lightweight parallel multi-scale feature extraction network to obtain a set of multimodal multi-scale feature vectors; S4, input the multimodal multiscale feature vector set into a dynamic gated fusion unit within a scale to generate a fused feature vector at that scale; S5, perform cross-scale feature aggregation on the fused feature vectors generated at all scales to generate a global fused feature descriptor; S6, the global fusion feature descriptor is input into a classifier, which calculates the final classification result of the multimodal data segment through a sequential linear layer and an activation function.

2. The lightweight multi-scale adaptive multimodal data fusion method according to claim 1, characterized in that, In step S1, the raw modal data stream includes at least two different data types, which are selected from video data stream, audio data stream, inertial measurement unit data stream, or radar point cloud data stream.

3. The lightweight multi-scale adaptive multimodal data fusion method according to claim 2, characterized in that, In step S3, the multi-scale feature extraction network sets up an independent feature extraction channel for each modality data, and each feature extraction channel contains at least two parallel feature extraction branches with different receptive field scales. The feature extraction branches use depthwise separable convolution operations to simultaneously extract deep feature maps of each modality at different spatiotemporal granularities. For the feature extraction channel of video data stream, its parallel feature extraction branches include a small-scale branch, a medium-scale branch and a large-scale branch; The small-scale branch uses a 3×3 convolutional kernel with a stride of 1; the medium-scale branch uses a 5×5 convolutional kernel with a stride of 2; the large-scale branch uses a 7×7 convolutional kernel with a stride of 2 after setting a max pooling layer at the input; all convolutional operations are depthwise separable convolutions, which are composed of a depthwise convolution and a point convolution, and a batch normalization layer and a non-linear activation function layer are configured after the convolutional layer; the feature maps output by each branch are used to generate video feature vectors of the corresponding scale after global average pooling; For the feature extraction channels of audio data stream, inertial measurement unit data stream or radar point cloud data stream, a multi-scale depth-separable convolution branch symmetrical to the video channel structure is used for feature extraction, and the feature vectors of the corresponding modes at each scale are output. Finally, the feature vectors of all modalities at all scales are combined into the multimodal multiscale feature vector set.

4. The lightweight multi-scale adaptive multimodal data fusion method according to claim 3, characterized in that, The specific process of step S4 is as follows: S41, For any scale, the feature vectors of all modes under that scale are input one by one into a gated weight generation network, wherein the gated weight generation network is a two-layer multilayer perceptron; S42, the first layer of the two-layer multilayer perceptron maps the input feature vector to a low-dimensional latent space, and the second layer maps the low-dimensional latent space representation to a scalar value; S43, apply a soft maximum function to normalize the contribution scores of all modalities to generate a set of dynamic fusion weights that sum to 1; S44, the feature vectors of the corresponding modal are weighted and summed using the dynamic fusion weights to generate the fusion feature vector at that scale.

5. The lightweight multi-scale adaptive multimodal data fusion method according to claim 4, characterized in that, The gated weight generation network adopts a shared parameter structure when processing feature fusion at different scales, forcing the network to learn a modality-independent value evaluation function based on the feature content itself.

6. The lightweight multi-scale adaptive multimodal data fusion method according to claim 5, characterized in that, The specific process of step S5 is as follows: S51, the fused feature vectors at each scale are concatenated along the feature dimension to form a high-dimensional concatenated vector; S52, the high-dimensional concatenated vector is input into a fully connected layer, the fully connected layer maps the input dimension to a preset fixed dimension, and outputs the global fusion feature descriptor.

7. A lightweight multi-scale adaptive multimodal data fusion method according to claim 6, characterized in that, The specific process of step S6 is as follows: S61, the global fusion feature descriptor is mapped to the dimension of the total number of classification categories through a linear layer; S62, Apply a soft maximum function to the mapping result to generate a probability vector, where each element of the probability vector represents the probability that the input data belongs to the corresponding category; S63, select the category corresponding to the maximum value in the probability vector as the final classification result.

8. The lightweight multi-scale adaptive multimodal data fusion method according to claim 1, characterized in that, The data preprocessing includes timestamp alignment, data frame segmentation, and normalization operations. The timestamp alignment is achieved through sliding window interpolation. Based on the timestamp of the video frame, linear interpolation or nearest neighbor interpolation is performed on other modal data to align all modal data to the same time grid in the time dimension. The data frame segmentation adopts a fixed time window mechanism. The normalization operation is performed separately for each modality, mapping the values ​​to the 0 to 1 interval or performing range normalization.

9. A lightweight multi-scale adaptive multimodal data fusion system, characterized in that, To implement the fusion method as described in claim 1 or 8, the method includes: The multimodal data acquisition and preprocessing module is used to capture raw modal data streams through multiple heterogeneous sensor interfaces, and perform timestamp alignment, data frame segmentation based on fixed time windows, and numerical normalization to output standardized multimodal data fragments. The lightweight parallel multi-scale feature extraction module is configured with an independent feature extraction channel for each modality, and each channel contains multiple parallel feature extraction branches. The scale-in-scale dynamic gating fusion module includes a gating weight generation network; The cross-scale feature aggregation module receives a set of fused feature vectors and integrates multi-scale information into a single, fixed-dimensional global fused feature descriptor by vector concatenation and mapping with a fully connected network. The task classification module receives the global fusion feature descriptor, performs calculations through a feedforward neural network classifier, and outputs the final classification decision result.

10. A lightweight multi-scale adaptive multimodal data fusion system according to claim 9, characterized in that, The gated weight generation network is a two-layer multilayer perceptron with shared parameters, used to uniformly evaluate the contribution of each modal feature at all scales.

Images