A deep learning-based multi-modal signal recognition method, device and medium

Abstract

The application provides a kind of multi-modal signal recognition method, equipment and medium based on deep learning, it is related to multi-modal signal recognition technical field.The method comprises: obtaining multiple modal original signals and pre-processing and space-time alignment;Multi-branch feature extraction network is constructed to extract the deep feature atlas of each modal signal;Deep feature atlas is fused into global fusion feature vector by attention mechanism;Based on global fusion feature vector, the recognition result is output by classification recognition network;Joint loss function is constructed and the weight of each loss is dynamically adjusted;Based on joint loss function, back propagation optimization is carried out, and the gradient update of each modal parameter is regulated by calculating the consistency coefficient in the optimization process;Part of modal data channel is randomly discarded with preset probability, and repeated training is carried out until convergence.The application can realize accurate identification of multi-modal signal, effectively suppress the gradient conflict between modes, and improve the robustness under the condition of modal loss.

Description

Technical Field

[0001] This invention relates to the field of multimodal signal recognition technology, and in particular to a multimodal signal recognition method, device and medium based on deep learning. Background Technology

[0002] With the rapid development of intelligent sensing, IoT, and AI technologies, the demand for signal monitoring and recognition in various scenarios continues to increase. Single-modal signals, due to their limited information dimensions and weak anti-interference capabilities, are insufficient to meet the high-precision recognition requirements of complex working conditions and environments. Multimodal signal fusion recognition technology can integrate multi-dimensional information such as audio, images, sensor waveforms, and environmental parameters, breaking through the information limitations of single-modal signals. It has become a core technology direction in fields such as intelligent recognition, fault diagnosis, scene perception, and safety monitoring.

[0003] In existing multimodal signal recognition technologies, traditional methods often employ shallow machine learning algorithms or simple feature concatenation and fusion, which can only extract shallow features and cannot uncover deep semantic associations within multimodal signals, resulting in low feature utilization efficiency. While some deep learning solutions introduce attention mechanisms for feature selection, most use fixed-weight fusion models, failing to dynamically adjust the fusion weights of each modality based on real-time signal quality. When a modal signal is distorted by interference, the interference information is forcibly injected into the fused features, directly lowering the overall recognition accuracy. Different modal signals exhibit significant differences in scale, dimension, and noise distribution. Existing technologies lack targeted modal denoising and adaptive weight allocation mechanisms, making them susceptible to single-modal noise interference, leading to redundant fused features and the loss of effective features.

[0004] Existing technologies also have significant shortcomings in multimodal joint training and optimization. Gradient directions generated by different modalities may conflict during backpropagation, leading to low optimization efficiency and difficulty in convergence. Most existing technologies employ a uniform optimization strategy, failing to adequately consider the optimization imbalance among modalities during training. Furthermore, the network training process does not effectively balance classification accuracy, feature fusion consistency, and modal differences, resulting in poor model convergence, insufficient recognition stability, weak adaptability to different operating conditions and noise levels, and limited model generalization ability. In practical applications, sensor failures, communication interruptions, and environmental interference often lead to the loss of some modal data. Existing technologies mostly assume that all modalities are always available during training and inference, lacking effective mechanisms to address modal data loss. When data for a particular modality is missing, the model often fails to function properly or experiences a sharp drop in recognition accuracy.

[0005] To address the aforementioned technical shortcomings, there is an urgent need for a high-precision, highly interference-resistant, and highly generalizable deep learning recognition scheme for multimodal signals. This scheme should possess the capability for full extraction and deep fusion of multimodal features, dynamically adjust fusion weights and optimization strategies based on the quality and uncertainty of each modality signal, effectively suppress single-modal noise interference and inter-modal gradient conflicts, and maintain stable recognition performance even under conditions where some modalities are missing, thereby solving the core pain points of existing technologies. Summary of the Invention

[0006] The purpose of this invention is to provide a method, device and medium for multimodal signal recognition based on deep learning. By extracting deep features from multiple branches, fusing cross-modal latent queries with cross-attention, dynamically weighting uncertainty and adjusting direction-aware gradients, it achieves high-precision and high-stability recognition of multimodal signals and improves the robustness and generalization ability of the model under modality missing conditions.

[0007] To achieve the above objectives, this invention provides a multimodal signal recognition method based on deep learning, comprising the following steps: Step S1: Obtain the original signals of multiple modalities in the scene to be identified, and perform preprocessing and spatiotemporal alignment on each original modal signal to obtain a standardized multimodal signal; Step S2: Based on standardized multimodal signals, construct a multi-branch feature extraction network to extract deep feature maps of each modality signal; Step S3: Based on the deep feature map, fuse it into a global fusion feature vector through an attention mechanism; Step S4: Based on the global fusion feature vector, output the recognition result through the classification and recognition network; Step S5: Construct a joint loss function based on the classification and recognition task, and dynamically adjust the weights of each loss term in the joint loss function; Step S6: Perform backpropagation optimization based on the joint loss function. Calculate the directional consistency coefficient during the backpropagation optimization process. Adjust the gradient update direction and magnitude of each modality parameter based on the directional consistency coefficient to update the network parameters. Step S7: Randomly discard the complete data channels of some modes in the standardized multimodal signal with a preset probability, and repeat steps S4 to S6 until the preset convergence condition is met and the iteration is terminated.

[0008] Preferably, in step S1, the multiple modal raw signals include audio signals, image and video signals, vibration sensing signals, and temperature and pressure sensing signals; Preprocessing includes filtering and noise reduction, outlier removal, and data normalization: The filtering and noise reduction employs an adaptive wavelet noise reduction algorithm, which determines the maximum number of wavelet decomposition levels based on the frequency distribution characteristics of the original signals for each mode. For the high-frequency detail coefficients obtained from each decomposition layer, a hard thresholding function is used to suppress high-frequency noise, while a soft thresholding function is used to preserve effective signal components for the low-frequency approximation coefficients. The thresholds for each layer are... Based on the noise standard deviation of this layer According to the formula Calculation, where For the first The length of the layer detail factor, For decomposition layer index, Noise standard deviation It is obtained by estimating the median absolute deviation of the detail coefficients at each level; Outlier removal uses the Raida criterion. When the deviation of each modal signal data point from its mean exceeds three times the standard deviation, the data point is determined to be an outlier and removed. The mean of the removed data is then used to fill the gap. Data normalization uses the min-max normalization algorithm to uniformly map each modal signal data to the [0, 1] interval; Spatiotemporal alignment includes: performing time-series alignment of preprocessed modal signals based on a timestamp synchronization mechanism, with a time-series alignment accuracy at the millisecond level; and completing spatial registration between pixel coordinates and physical coordinates for image signals and sensor array signals.

[0009] Preferably, in step S2, the multi-branch feature extraction network sets up an independent single-modal feature extraction sub-network for each modal signal. Each sub-network adopts a structure combining convolutional neural networks and residual networks. It extracts shallow texture, waveform, and pixel detail features of the signal through multiple convolutional layers, and mines deep semantic association features through residual block stacking, and outputs deep feature maps of each modality. Each sub-network output introduces a channel attention mechanism to perform weight selection on the deep feature maps of each modality. The channel attention mechanism generates the channel weights of each feature through global pooling, weight mapping of fully connected layers, and sigmoid activation function, thereby obtaining refined feature maps of each modality.

[0010] Preferably, in step S3, the attention mechanism is a cross-modal latent query cross-attention fusion network, which aggregates features in the following way: A cross-modal latent query cross-attention fusion network initializes a set of modality-independent learnable latent query vectors as the query end, and uses the refined feature maps of each modality as the key end. The learnable latent query vectors and the refined feature maps of each modality are input into the multi-head cross-attention network, and the features of each modality are aggregated in a unified semantic feature subspace to generate a global fusion feature vector. The number of learnable potential query vectors is a preset fixed value, and its dimension matches the output dimension of the refined feature maps of each modality; the multi-head cross-attention network includes One's attention, As a preset positive integer, each attention head independently calculates the attention distribution between the learnable latent query vector and the refined features of each modality. The outputs of each attention head are concatenated and linearly mapped to generate the final fused feature; the learnable latent query vector is updated synchronously with the network parameters during model training.

[0011] Preferably, in step S5, the joint loss function is constructed in the following way: Based on the classification and recognition task, a joint loss function is constructed to build a dual uncertainty evidence network, which simultaneously models the random uncertainty and cognitive uncertainty in the multimodal signal recognition task; The randomness and uncertainty are modeled by adding a Gaussian noise layer to the output of each modality feature extraction subnetwork, and the heteroscedasticity noise parameters of each modality are learned using maximum likelihood estimation. ,in For modal indexing, , The total number of modes; Cognitive uncertainty is approximated by the Monte Carlo Dropout method, which calculates the variance of classification probabilities in multiple forward propagations; The formula for calculating the joint loss function is: ; in, For the joint loss function value, For classification cross-entropy loss, For feature fusion consistency loss, For modal difference loss, , , These are the learnable noise parameters corresponding to classification loss, fusion consistency loss, and modality difference loss, respectively. , , These represent the weights of classification loss, fusion consistency loss, and modality difference loss, respectively; the loss weights for each modality. ,in These are the heteroscedasticity noise parameters for each mode; The weights of each loss term in the joint loss function are dynamically adjusted based on randomness and uncertainty. Random uncertainty also serves as a priori basis for gradient direction control, based on heteroscedasticity noise parameters. ,by The gradient direction corresponding to the smallest mode is used as the gradient direction of the global fusion feature. A reliable reference direction.

[0012] Preferably, in step S6, the directional consistency coefficient is calculated and used for gradient control in the following manner: In each backpropagation iteration, the modal gradient direction generated by each modal feature extraction subnetwork is calculated. With global fusion feature gradient direction Cosine similarity between ; Calculating the directional consistency coefficient based on cosine similarity : ; in, For linear rectified activation function, when When <0, =0, indicating that the modal gradient conflicts with the global optimization objective in terms of direction; The directional consistency coefficient With modal weights Multiplying them together yields the integrated gradient weights for each mode. ; Calculate the integrated gradient : ; To integrate gradients Update all optimizable parameters of the multi-branch feature extraction network, the cross-modal latent query cross-attention fusion network, and the classification and recognition network.

[0013] Preferably, in step S7, the preset convergence condition is: At the end of each training cycle, calculate the orientation consistency coefficients of each mode obtained in step S6 across all iterations within the current cycle. The variance when continuous The iteration terminates when the variance within a training cycle is less than a preset variance threshold, the decrease in the joint loss function value obtained in step S5 is less than a preset loss threshold, and the cognitive uncertainty obtained in step S5 is less than a preset confidence threshold. It is a preset positive integer, with a value range of 3 to 10.

[0014] Preferably, in step S7, the preset probability ranges from 0.1 to 0.5, and a linear decay strategy is adopted during training to gradually decay from the initial value to the final value. Simultaneously, a knowledge distillation strategy is adopted, using the network output under complete modal input as a soft label to guide the network output under modal missing conditions to approach the soft label. The distillation loss adopts the KL divergence loss function.

[0015] The present invention also provides a deep learning-based multimodal signal recognition device, comprising: The signal acquisition and preprocessing module is used to acquire multiple modal raw signals in the scene to be identified, and to preprocess and align the original signals of each modality in time and space to obtain standardized multimodal signals. The multi-branch feature extraction module is used to construct a multi-branch feature extraction network based on standardized multimodal signals and extract deep feature maps of each modality signal; The cross-modal feature fusion module is used to fuse deep feature maps into a global fusion feature vector through an attention mechanism; The classification and recognition module is used to output recognition results through a classification and recognition network based on the global fusion feature vector. The joint loss construction module is used to construct a joint loss function based on the classification and recognition task, and the weights of each loss term in the joint loss function are dynamically adjusted. The gradient control optimization module is used for backpropagation optimization based on the joint loss function. During the backpropagation optimization process, the directional consistency coefficient is calculated, and the gradient update direction and update magnitude of each modality parameter are adjusted based on the directional consistency coefficient to update the network parameters. The modality discarding and iterative training module is used to randomly discard the complete data channels of some modalities in the standardized multimodal signal with a preset probability, and repeatedly execute the classification and recognition, joint loss construction and gradient control optimization process until the preset convergence condition is met and the iteration is terminated.

[0016] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described deep learning-based multimodal signal recognition method.

[0017] Therefore, the present invention employs the above-mentioned deep learning-based multimodal signal recognition method, device, and medium, and the beneficial technical effects are as follows: (1) This invention uses a cross-modal latent query cross-attention fusion network to adaptively aggregate modal features in a unified semantic feature subspace with learnable latent query vectors independent of modality type. This solves the problem of modal information fragmentation caused by feature splicing or fixed weight fusion in the prior art and improves the expression quality and recognition accuracy of fused features.

[0018] (2) By combining random uncertainty dynamic weighting with direction-aware gradient control, the present invention synchronously adjusts the weight of the loss function and suppresses the modal gradients of directional conflicts during backpropagation, thus solving the optimization imbalance and convergence difficulties caused by fixed weight loss function and intermodal gradient conflicts in the prior art, and improving the convergence effect and optimization stability of model training.

[0019] (3) This invention randomly discards some modal data channels with a preset probability and performs joint training by combining knowledge distillation, so that the model can still use the remaining modal features to complete the recognition under any modal missing conditions. This solves the problem of decreased recognition accuracy caused by the lack of modal missing response mechanism in the prior art, and enables the model to maintain stable recognition performance when facing modal missing in actual deployment. Attached Figure Description

[0020] Figure 1 This is a flowchart of a deep learning-based multimodal signal recognition method according to the present invention; Figure 2 This is a graph showing the change in joint loss value versus classification accuracy during the training process. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 This embodiment provides a deep learning-based multimodal signal recognition method applied to industrial equipment operation status monitoring and fault identification scenarios. Industrial equipment generates multiple modal signals during operation, including vibration signals, audio signals, temperature signals, and image signals. A single modal signal is insufficient to comprehensively reflect the equipment's operating status. This embodiment achieves accurate identification of equipment operating status and fault diagnosis by fusing these four modal signals.

[0024] Dataset and experiment configuration.

[0025] This embodiment uses a dataset of equipment operating status collected from an industrial site, including five states: normal operation, bearing failure, gear failure, circuit failure, and overheating failure. Data acquisition utilizes an industrial IoT sensing system, simultaneously collecting four modalities of data: vibration sensing signals, operating noise audio signals, equipment surface temperature sensing signals, and equipment operating screen image signals. The dataset contains a total of 12,000 samples, with 9,600 samples in the training set and 2,400 samples in the test set.

[0026] In this embodiment, both model training and testing were performed on a deep learning workstation equipped with an Intel Xeon Gold 6248 processor, two NVIDIA Tesla V100 graphics cards (32GB VRAM each), an Ubuntu 20.04 LTS operating system, and PyTorch 1.12.0 with CUDA version 11.6 as the deep learning framework. During model training, the initial learning rate was set to... The Adam optimizer is used for parameter updates, the learning rate decay strategy is cosine annealing, the number of training epochs is set to 200, and the batch size is 64.

[0027] The overall process of the method in this embodiment is as follows: Figure 1 As shown.

[0028] Step S1: Obtain the original signals of multiple modalities in the scene to be identified, preprocess and align the original signals of each modality in time and space to obtain standardized multimodal signals.

[0029] In this step, the four modal raw signals during equipment operation are first collected synchronously using an industrial IoT sensing system: (1) Vibration sensing signal: A piezoelectric accelerometer is installed at the bearing seat of the equipment. The sampling frequency is 12.8kHz, the sampling time is 2 seconds, and 25,600 sampling points are obtained.

[0030] (2) Operating noise audio signal: A high-fidelity microphone is installed 1 meter in front of the device, with a sampling frequency of 44.1kHz and a sampling duration of 2 seconds each time.

[0031] (3) Equipment surface temperature sensing signal: PT100 platinum resistance temperature sensor is installed at key measuring points on the equipment shell, with a sampling frequency of 1Hz and a sampling duration of 2 seconds each time.

[0032] (4) Equipment operation screen image signal: An industrial camera with a resolution of [missing information] is installed on the side of the equipment. Pixels, frame rate 30fps, 10 frames of images captured each time.

[0033] After the data acquisition is completed, the raw signals of each mode are preprocessed.

[0034] The filtering and noise reduction process employs an adaptive wavelet noise reduction algorithm. Taking vibration signals as an example, based on the frequency distribution characteristics of the vibration signals, their main energy is concentrated in the range of 0.5kHz to 5kHz, and the maximum number of wavelet decomposition levels is determined accordingly. =5. The signal is decomposed into 5 levels using the db4 wavelet basis function to obtain the high-frequency detail coefficients at each level. and low-frequency approximation coefficients The noise standard deviation was calculated for each layer of detail coefficients. The median absolute deviation estimation method is adopted. Substitute into the formula Calculate the threshold for each layer, where For the first The length of the detail factor. When the absolute value of the detail factor is greater than or equal to... When the coefficient is less than 1, it is retained. Setting the time threshold to zero enables hard threshold noise reduction, effectively suppressing high-frequency random noise. A soft threshold function is used to process low-frequency approximation coefficients, preserving the main characteristic components of the equipment's operating state. After noise reduction, the signal-to-noise ratio of the vibration signal increases from the original 10.2dB to 24.6dB, effectively extracting fault feature information submerged in strong background noise.

[0035] The filtering and noise reduction methods for audio signals are similar to those for vibration signals, the difference being that audio signals have a wider frequency range, with their main energy concentrated in the 0.1kHz to 8kHz range. The maximum number of wavelet decomposition levels is set to... =6, using the db6 wavelet basis function to better match the time-frequency characteristics of the audio signal.

[0036] Outlier removal uses the Raida criterion (3) (Criteria). Taking temperature sensor signals as an example, calculate the mean value of temperature data within the acquisition period. and standard deviation When the temperature value at a certain sampling point exceeds ±3 When the range is defined, the data point is determined to be an outlier, removed, and the mean of the remaining data is used to fill the gap. In this embodiment, due to occasional interference from the sensor, the outliers removed account for approximately 0.15% of the total data volume in the temperature signal collected.

[0037] Data normalization employs the min-max normalization algorithm, which maps the amplitude data of vibration signals, audio signals, and temperature signals to the [0,1] interval, and maps the pixel values ​​of image signals from [0,255] to the [0,1] interval, thereby eliminating the differences in dimensions and scales between different modal signals.

[0038] Spatiotemporal alignment uses the device system clock as a unified time reference to perform time-series alignment of the four modal signals. The timing alignment accuracy is at the millisecond level, ensuring that vibration, audio, temperature, and image signals acquired at the same time are completely matched in the time dimension. Based on the spatial correspondence between vibration sensor points and image pixel coordinates, spatial registration is completed between the vibration measurement point coordinates and image pixel coordinates, achieving spatial unification of multimodal signals.

[0039] Step S2: Based on standardized multimodal signals, construct a multi-branch feature extraction network to extract deep feature maps of each modality signal.

[0040] A four-branch feature extraction network is constructed, with independent single-modal feature extraction subnetworks set up for each of the four modes: vibration, audio, temperature, and image.

[0041] The vibration signal branch employs a structure combining a one-dimensional convolutional neural network and a residual network. First, three one-dimensional convolutional layers are used to extract the time-domain waveform features of the vibration signal, with kernel sizes of 16, 8, and 4, and channel numbers of 64, 128, and 256, respectively, all with a stride of 1. The first convolutional layer extracts the basic waveform morphology and amplitude distribution features of the signal; the second layer extracts the frequency component variation features; and the third layer extracts the joint time-frequency features. Subsequently, four residual blocks are stacked, each containing two convolutional layers and skip connections, to mine deeper semantic features in the vibration signal corresponding to the equipment's operating state, such as the harmonic components of the rotation frequency and its sideband structure. The skip connections in the residual blocks effectively avoid the vanishing gradient problem in deep networks, enabling the network to stably train deeper features.

[0042] The audio signal branch adopts a one-dimensional convolutional residual network structure similar to that of the vibration signal branch, with convolutional kernel sizes of 16, 8, and 4, and channel numbers of 64, 128, and 256, respectively. Multi-layer convolutional layers extract the temporal waveform and spectral envelope features of the audio signal, corresponding to the loudness changes and timbre features of equipment operating noise. Stacked residual blocks further mine deeper semantic features reflecting changes in equipment operating status, such as the frequency distribution patterns of abnormal noise.

[0043] The temperature signal branch is a low-frequency, slowly changing signal. A relatively shallow convolutional network structure is adopted, which contains two one-dimensional convolutional layers with kernel sizes of 8 and 4, and channel numbers of 32 and 64, respectively, to extract the temporal variation trend and absolute value distribution characteristics of the temperature signal.

[0044] The image signal branch employs a structure combining a two-dimensional convolutional neural network and a residual network. First, four two-dimensional convolutional layers are used to extract the spatial texture and structural features of the image, with convolutional kernels of uniform size... The number of channels are 32, 64, 128, and 256, with a step size of 2. Then, four residual blocks are stacked, each containing two layers. Convolutional layers and skip connections are used to extract deep semantic features from the device's operation screen, such as changes in device appearance and indicator light status. The output feature maps of all branches are uniformly sized to 256-dimensional feature vectors.

[0045] Each branch network output incorporates a channel attention mechanism (SE attention module). Taking the vibration signal branch as an example, global average pooling is performed on the output 256-dimensional feature map, compressing the global information of each feature channel into a scalar value, resulting in a 256-dimensional channel description vector. Two fully connected layers are used to weight the channel description vector. The first fully connected layer compresses the 256 dimensions to 64 dimensions, and the second fully connected layer restores the 64 dimensions to 256 dimensions, introducing ReLU and Sigmoid activation functions to generate 256 channel weight values ​​between 0 and 1. These generated channel weight values ​​are multiplied channel-by-channel with the original feature map to recalibrate the feature channels, strengthening those that contribute significantly to fault identification, such as the frequency band corresponding to the fault characteristic frequency in the vibration signal, while suppressing invalid feature channels containing noise interference, resulting in a refined feature map of the vibration modes. The same processing flow is used for the channel attention mechanisms in the audio, temperature, and image branches.

[0046] This step combines multi-branch independent feature extraction with channel attention filtering to fully explore both shallow detail features and deep semantic features of each modality, while effectively filtering out invalid noise features in each modality. This provides high-quality, refined feature input for subsequent cross-modal fusion, ensuring recognition performance from the source.

[0047] Step S3: Based on the deep feature map, fuse it into a global fusion feature vector through an attention mechanism.

[0048] In this step, the attention mechanism is specifically a cross-modal latent query cross-attention fusion network.

[0049] Initialize a set of 64 learnable latent query vectors independent of modality type as query endpoints. Each query vector has a dimension of 256, which matches the output dimension of the refined feature map of each modality.

[0050] The refined feature maps of the four modes obtained in step S2 are used together as key-value pairs. The feature maps of vibration, audio, temperature, and image modes are all 256-dimensional. After stacking, they form a dimensional... The key-value matrix is ​​used to generate the 64 learnable latent query vectors. These 64 vectors are then fed into a multi-head cross-attention network. The network contains eight attention heads, each independently calculating the attention distribution between the 64 query vectors and four modal features. This involves calculating the relevance score between each query vector and each modal feature, normalizing it using Softmax to obtain the attention weights, and then weighting and summing the results for each modal feature. The outputs of the eight attention heads are concatenated and linearly mapped to generate... The intermediate fusion features are then subjected to global average pooling to obtain a 256-dimensional global fusion feature vector.

[0051] The aforementioned learnable potential query vectors are updated synchronously with the network parameters during model training, gradually optimizing their ability to aggregate multimodal information. Through this mechanism, the model can adaptively aggregate effective information from each modality within a unified semantic feature subspace based on the actual contribution of each modality's features. In this embodiment, after training, the attention weight allocation ratio for each modality is approximately 30% for vibration, 28% for audio, 18% for temperature, and 24% for image. Vibration and audio signals contribute the most to fault identification, while temperature signals contribute relatively less.

[0052] This step utilizes a cross-attention mechanism guided by learnable latent query vectors, abandoning the rigid fusion mode of traditional fixed-weight concatenation. This allows the fusion weights to adaptively evolve during training based on the actual contribution of each modality's features. Since the latent query vectors are independent of any specific modality, the model can fairly compare and flexibly aggregate information from different modalities within a unified semantic space, fully mining deep complementary relationships between modalities and fundamentally solving the problem of fragmented modal information.

[0053] Step S4: Based on the global fusion feature vector, output the recognition result through the classification and recognition network.

[0054] The 256-dimensional global fusion feature vector obtained in step S3 is input into the classification and recognition network. The classification and recognition network adopts a two-layer fully connected layer structure. The first fully connected layer maps the 256 dimensions to 128 dimensions and uses the ReLU activation function and Dropout (dropout rate 0.5) to prevent overfitting. The second fully connected layer maps the 128 dimensions to 5 dimensions and outputs the probability distribution of the device operating status belonging to five categories: normal operation, bearing failure, gear failure, circuit failure, and overheating failure after passing through the Softmax activation function. The category with the highest output probability value is taken as the final recognition result.

[0055] Step S5: Construct a joint loss function based on the classification and recognition task, and dynamically adjust the weights of each loss term in the joint loss function.

[0056] Based on the classification and recognition task in step S4, a joint loss function is constructed.

[0057] We construct a dual uncertainty evidence network to simultaneously model stochastic uncertainty and cognitive uncertainty in multimodal signal recognition tasks.

[0058] The randomness and uncertainty are modeled by adding a Gaussian noise layer to the output of each modal feature extraction subnetwork in step S2. Taking the vibration signal branch as an example, a Gaussian noise layer is added to the output of the vibration feature extraction subnetwork, and the standard deviation of this noise layer is... As a learnable parameter, the initial value is set to 1.0. This parameter is learned during training using maximum likelihood estimation. When the vibration signal quality is good and contains rich fault feature information, the network automatically... The learning value is relatively small. After training is completed in this embodiment... The loss weight is approximately 0.32, assigning a high weight to this mode. The temperature signal branch, audio signal branch, and image signal branch also add Gaussian noise layers to the output of their respective feature extraction sub-networks. The standard deviation of each noise layer is denoted as... , , This parameter also participates in training as a learnable parameter. After training is completed in this embodiment, Approximately 2.18, Approximately 0.41, It is approximately 0.87.

[0059] Cognitive uncertainty is approximated by the Monte Carlo Dropout method. During the testing phase, the same input sample is forward-propagated 20 times, and different neurons are randomly dropped each time. The variance of the classification probability is calculated for the 20 times. The larger the variance, the more uncertain the model's prediction result for the sample, and the lower the confidence level.

[0060] The formula for calculating the joint loss function is: ; in, For the joint loss function value, For classification cross-entropy loss, The feature fusion consistency loss measures the distribution difference between the globally fused features and the features of each modality, and is calculated using KL divergence. To address modal difference loss and promote the learning of unique features by different modalities, a central loss method is used for calculation. , , These are the learnable noise parameters corresponding to the classification loss, fusion consistency loss, and modality difference loss, respectively. The weights of the classification loss are... Weights of consistency loss Weights of modal difference loss .

[0061] The weights of each loss term in the joint loss function are dynamically adjusted based on the randomness and uncertainty. After training is completed in this embodiment, Approximately 0.25 Approximately 0.73 Approximately 0.95, with corresponding weights as follows: Approximately 8.0 Approximately 0.94 The loss is approximately 0.55, indicating that classification loss dominates, which is consistent with the core objective of classification and recognition tasks.

[0062] Meanwhile, the heteroscedasticity noise parameters of each mode in random uncertainty ( (Vibration, audio, temperature, and image) simultaneously serve as prior information for gradient direction control, in order to The gradient direction corresponding to the smallest mode, i.e., the vibration mode in this embodiment, is used as the gradient direction of the global fusion feature. A reliable reference direction.

[0063] This step models both stochastic and cognitive uncertainty simultaneously using a dual uncertainty evidence network, enabling the model to distinguish between two different sources of uncertainty: noise in the data itself and the model's uncertainty about the prediction. Based on a dynamic weighting mechanism for stochastic uncertainty, the weights of each component in the loss function are automatically adjusted according to the real-time signal quality of each modality—high-quality, information-rich modes receive higher optimization priority, while degraded, information-poor modes automatically reduce their contribution. This overcomes the limitations of traditional methods that rely on manual trial-and-error weight adjustments, making the training process more stable and efficient.

[0064] Step S6: Perform backpropagation optimization based on the joint loss function. Calculate the directional consistency coefficient during the backpropagation optimization process. Adjust the gradient update direction and magnitude of each modality parameter based on the directional consistency coefficient to update the network parameters.

[0065] In each backpropagation iteration, the modal gradient direction generated by each modal feature extraction subnetwork in step S2 is first calculated. ( (Corresponding to the four branches of vibration, audio, temperature, and image respectively) and the global fusion feature gradient direction Cosine similarity between .

[0066] Calculating the directional consistency coefficient based on cosine similarity .when When ≥0, Equal to the cosine similarity value, with a numerical range of 0 to 1; when When <0, =0 indicates a directional conflict between the modal gradient and the global optimization objective. In this embodiment, gradient conflicts are relatively frequent in the first 30 iterations of training, occurring in approximately 15% to 20% of the iterations for a particular modality. When the gradient is 0, the temperature mode experiences the highest frequency of directional conflicts due to its relatively low information content. As training progresses, the gradient directions of each mode gradually become more consistent, and the directional conflict phenomenon basically disappears after the later 150 rounds of training.

[0067] The directional consistency coefficient The modal weights determined by dynamic weighting based on uncertainty in step S5 Multiplying them together yields the integrated gradient weights for each mode. When the gradient direction of a certain mode is consistent with the global optimization direction, that is... The value is relatively large, and the uncertainty of this mode is low, i.e. When the gradient weights are large, integrate the gradient weights. When the maximum value is reached, this mode plays a dominant role in parameter updates; when a conflict occurs... =0, the gradient of this mode is completely suppressed.

[0068] Calculate the integrated gradient This embodiment integrates all optimizable parameters of the feature extraction network of the four branches in gradient update step S2, the cross-modal latent query cross-attention fusion network in step S3, and the classification and recognition network in step S4. The Adam optimizer is used in this embodiment, with an initial learning rate of... The learning rate was decayed by multiplying it by 0.5 every 60 epochs, for a total of 200 epochs. Training terminated early when the convergence condition was met in the 178th epoch, effectively avoiding overfitting and saving training time.

[0069] This step effectively solves the core problem of conflicting gradient directions among different modalities in multimodal joint training by calculating the directional consistency coefficient and adjusting the gradient. The directional consistency coefficient, as a soft gating signal, works synergistically with the modal weights determined by the dynamic weighting of uncertainty in step S5 to achieve selective parameter updates for each modal gradient. This ensures that the overall optimization objective of the model remains consistent across modalities, significantly improving the convergence efficiency and final performance of multimodal joint training.

[0070] Step S7: Randomly discard the complete data channels of some modes in the standardized multimodal signal with a preset probability, and repeat steps S4 to S6 until the preset convergence condition is met and the iteration is terminated.

[0071] In each training iteration, the standardized multimodal signal obtained in step S1 is processed with a preset probability. Randomly discard complete data channels of some modalities, adding mask markers at the discard locations, and setting all data of the discarded modalities to zero. Preset probability. Setting the threshold to 0.3 at the beginning of training means that approximately 30% of the samples in each batch of training data will randomly lack complete data for one modality. The selection of the missing modality is completely random; all four modalities (vibration, audio, temperature, and image) may be discarded. Taking the vibration mode being discarded as an example, the sample is identified based solely on the features of the audio, temperature, and image modalities. The model is forced to learn to perform fault classification without relying on the vibration signal, making full use of the redundant information in the audio and image.

[0072] Preset probability A linear decay strategy was employed during training, gradually decreasing the value from an initial 0.3 to a final 0.05, specifically by approximately 0.05 every 40 rounds. This allowed the model to transition from early forced learning of redundant multimodal representations to later fine-tuning. The higher dropout rate in the early training phase forced the model to fully utilize redundant information from all available modalities, avoiding over-reliance on any particular modality. The lower dropout rate in the later training phase allowed for fine-tuning of the model with all modalities intact, further improving recognition accuracy.

[0073] Simultaneously, a knowledge distillation strategy is employed, using the network output under complete four-modal input in each training iteration as a soft label. This guides the network output under modality-missing conditions to approximate this soft label, with the distillation loss employing the KL divergence loss function. The complete teacher network output contains rich inter-category correlation information, such as the feature similarity between bearing faults and gear faults, guiding the student network to learn this fine-grained discriminative information under modality-missing conditions. The weight coefficient of the distillation loss is set to 0.3, and it is weighted and summed with the classification loss to obtain the total loss.

[0074] Repeat steps S4 to S6. After each training cycle, traverse the entire training set once and calculate the consistency coefficients of each modal orientation obtained in step S6 for all iterations in the current cycle. The variance of the directional consistency coefficient is considered to be less than a preset variance threshold of 0.01, the decrease in the joint loss function value is less than a preset loss threshold of 0.005, and the cognitive uncertainty is less than a preset reliability threshold of 0.15 within eight consecutive training cycles. In this embodiment, the above three-dimensional convergence conditions are met for three consecutive cycles: 178th, 179th, and 180th cycles, and the model terminates training early in the 180th cycle.

[0075] This step involves randomly discarding modal data channels during training, exposing the model to a large number of samples with missing modalities during the training phase. This allows the model to learn to achieve accurate recognition even under conditions of missing modalities, utilizing the remaining modalities. Simultaneously, a knowledge distillation strategy ensures that the network output under modal-missing conditions approximates the output under complete modal conditions, preserving the fine-grained inter-class discriminative information learned under complete modalities. A decay strategy with a dropout rate decreasing during training allows the model to first establish robustness in modal redundancy representations before fine-tuning, balancing robustness and accuracy. This ensures the model maintains stable recognition performance even when facing various sensor failures or data loss in real-world deployments.

[0076] The curves showing the change in joint loss function value and classification accuracy during training are as follows: Figure 2 As shown in the figure, the horizontal axis represents the number of training epochs, the left vertical axis represents the joint loss value, and the right vertical axis represents the classification accuracy. The figure shows that the joint loss value decreases most rapidly in the early stages of training, slows down after about 50 epochs, and stabilizes after about 120 epochs. The classification accuracy shows the most significant improvement in the early stages of training, decreases after about 50 epochs, and saturates after about 120 epochs. The trends of the two curves correspond well, indicating that the model can quickly learn effective features and gradually converge to a stable state during training.

[0077] The results of testing show that the method in this embodiment can identify five operating states of industrial equipment, as shown in Table 1.

[0078] Table 1. Identification results of each operating status

[0079] The method in this embodiment achieved an overall recognition accuracy of 97.8% on 2400 test samples, with recognition accuracy for each category exceeding 96.9%.

[0080] To verify the effectiveness of the method in this embodiment, various comparative methods were set up for experiments, and the results are shown in Table 2.

[0081] Table 2 Comparison of recognition accuracy of different methods

[0082] The comparative results show that the recognition accuracy of single-modal signals is much lower than that of multimodal fusion methods. Temperature signals, due to their limited information dimension, exhibit the worst recognition performance, validating the necessity of multimodal fusion. In traditional multimodal fusion methods, feature splicing fusion and fixed-weight attention fusion achieve recognition accuracies of 89.4% and 91.2%, respectively, both lower than the 97.8% of our proposed method. Through an uncertainty-based dynamic weighting mechanism, our method automatically adjusts the fusion weights based on the quality of each modal signal, effectively suppressing interference from low-quality modalities. The direction-aware gradient control mechanism effectively alleviates gradient conflicts between modalities, improving the convergence efficiency and stability of model training. Furthermore, the joint training strategy of dynamic modality discarding and knowledge distillation enhances the robustness of the model under modality loss conditions. These comparative experiments fully demonstrate the significant advantages of our proposed method in terms of recognition accuracy.

[0083] The changes in the directional consistency coefficients of each modality during training reflect the effectiveness of the gradient control mechanism. In the early stages of training, the directional consistency coefficients fluctuate significantly: vibration is approximately 0.72, audio is approximately 0.68, image is approximately 0.58, and temperature is approximately 0.41. The temperature modality, due to its lower information content, exhibits the lowest gradient direction consistency with other modalities and experiences the most frequent directional conflicts. As training progresses, the feature representations of each modality gradually harmonize, and the directional consistency coefficients continuously improve. By the 100th epoch, vibration is approximately 0.89, audio is approximately 0.86, image is approximately 0.81, and temperature is approximately 0.72. By the 178th epoch, the directional consistency coefficients of all modalities have reached above 0.90, indicating that the gradient directions of each modality tend to be consistent, and the model optimization has entered a stable state.

[0084] Example 2 A deep learning-based multimodal signal recognition device includes: The signal acquisition and preprocessing module is used to acquire multiple modal raw signals in the scene to be identified, and to preprocess and align the original signals of each modality in time and space to obtain standardized multimodal signals. The multi-branch feature extraction module is used to construct a multi-branch feature extraction network based on standardized multimodal signals and extract deep feature maps of each modality signal; The cross-modal feature fusion module is used to fuse deep feature maps into a global fusion feature vector through an attention mechanism; The classification and recognition module is used to output recognition results through a classification and recognition network based on the global fusion feature vector. The joint loss construction module is used to construct a joint loss function based on the classification and recognition task, and the weights of each loss term in the joint loss function are dynamically adjusted. The gradient control optimization module is used for backpropagation optimization based on the joint loss function. During the backpropagation optimization process, the directional consistency coefficient is calculated, and the gradient update direction and update magnitude of each modality parameter are adjusted based on the directional consistency coefficient to update the network parameters. The modality discarding and iterative training module is used to randomly discard the complete data channels of some modalities in the standardized multimodal signal with a preset probability, and repeatedly execute the classification and recognition, joint loss construction and gradient control optimization process until the preset convergence condition is met and the iteration is terminated.

[0085] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0086] Therefore, the present invention employs the above-mentioned method, device and medium for multimodal signal recognition based on deep learning, which can effectively suppress single-modal noise interference and intermodal gradient conflicts, improve the robustness and generalization ability of the model under modality missing conditions, and is applicable to the fields of intelligent monitoring, fault diagnosis, scene recognition and other technical fields.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal signal recognition method based on deep learning, characterized in that, Includes the following steps: Step S1: Obtain the original signals of multiple modalities in the scene to be identified, and perform preprocessing and spatiotemporal alignment on each original modal signal to obtain a standardized multimodal signal; Step S2: Based on standardized multimodal signals, construct a multi-branch feature extraction network to extract deep feature maps of each modality signal; Step S3: Based on the deep feature map, fuse it into a global fusion feature vector through an attention mechanism; Step S4: Based on the global fusion feature vector, output the recognition result through the classification and recognition network; Step S5: Construct a joint loss function based on the classification and recognition task, and dynamically adjust the weights of each loss term in the joint loss function; Step S6: Perform backpropagation optimization based on the joint loss function. Calculate the directional consistency coefficient during the backpropagation optimization process. Adjust the gradient update direction and magnitude of each modality parameter based on the directional consistency coefficient to update the network parameters. Step S7: Randomly discard the complete data channels of some modes in the standardized multimodal signal with a preset probability, and repeat steps S4 to S6 until the preset convergence condition is met and the iteration is terminated.

2. The deep learning-based multimodal signal recognition method according to claim 1, characterized in that, In step S1, the multiple modal raw signals include audio signals, image and video signals, vibration sensing signals, and temperature and pressure sensing signals; Preprocessing includes filtering and noise reduction, outlier removal, and data normalization: The filtering and noise reduction employs an adaptive wavelet noise reduction algorithm, which determines the maximum number of wavelet decomposition levels based on the frequency distribution characteristics of the original signals for each mode. For the high-frequency detail coefficients obtained from each decomposition layer, a hard thresholding function is used to suppress high-frequency noise, while a soft thresholding function is used to preserve effective signal components for the low-frequency approximation coefficients. The thresholds for each layer are... Based on the noise standard deviation of this layer According to the formula Calculation, where For the first The length of the layer detail factor, For decomposition layer index, Noise standard deviation It is obtained by estimating the median absolute deviation of the detail coefficients at each level; Outlier removal uses the Raida criterion. When the deviation of each modal signal data point from its mean exceeds three times the standard deviation, the data point is determined to be an outlier and removed. The mean of the removed data is then used to fill the gap. Data normalization uses the min-max normalization algorithm to uniformly map each modal signal data to the [0, 1] interval; Spatiotemporal alignment includes: performing time-series alignment of preprocessed modal signals based on a timestamp synchronization mechanism, with a time-series alignment accuracy at the millisecond level; and completing spatial registration between pixel coordinates and physical coordinates for image signals and sensor array signals.

3. The deep learning-based multimodal signal recognition method according to claim 1, characterized in that, In step S2, the multi-branch feature extraction network sets up an independent single-modality feature extraction sub-network for each modal signal. Each sub-network adopts a structure combining convolutional neural networks and residual networks. It extracts shallow texture, waveform, and pixel detail features of the signal through multiple convolutional layers, and mines deep semantic association features through residual block stacking, outputting deep feature maps of each modality. Each sub-network output introduces a channel attention mechanism to perform weight selection on the deep feature maps of each modality. The channel attention mechanism generates the channel weights of each feature through global pooling, weight mapping of fully connected layers, and sigmoid activation function, thereby obtaining refined feature maps of each modality.

4. The deep learning-based multimodal signal recognition method according to claim 1, characterized in that, In step S3, the attention mechanism is a cross-modal latent query cross-attention fusion network, which aggregates features in the following way: A cross-modal latent query cross-attention fusion network initializes a set of modality-independent learnable latent query vectors as the query end, and uses the refined feature maps of each modality as the key end. The learnable latent query vectors and the refined feature maps of each modality are input into the multi-head cross-attention network, and the features of each modality are aggregated in a unified semantic feature subspace to generate a global fusion feature vector. The number of learnable potential query vectors is a preset fixed value, and its dimension matches the output dimension of the refined feature maps of each modality; the multi-head cross-attention network includes One's attention, As a preset positive integer, each attention head independently calculates the attention distribution between the learnable latent query vector and the refined features of each modality. The outputs of each attention head are concatenated and then linearly mapped to generate the final fused feature. The learnable latent query vector is updated synchronously with the network parameters during model training.

5. The multimodal signal recognition method based on deep learning according to claim 1, characterized in that, In step S5, the joint loss function is constructed as follows: Based on the classification and recognition task, a joint loss function is constructed to build a dual uncertainty evidence network, which simultaneously models the random uncertainty and cognitive uncertainty in the multimodal signal recognition task; The randomness and uncertainty are modeled by adding a Gaussian noise layer to the output of each modality feature extraction subnetwork, and the heteroscedasticity noise parameters of each modality are learned using maximum likelihood estimation. ,in For modal indexing, , The total number of modes; Cognitive uncertainty is approximated by the Monte Carlo Dropout method, which calculates the variance of classification probabilities in multiple forward propagations; The formula for calculating the joint loss function is: ； in, For the joint loss function value, For classification cross-entropy loss, For feature fusion consistency loss, For modal difference loss, , , These are the learnable noise parameters corresponding to classification loss, fusion consistency loss, and modality difference loss, respectively. , , These represent the weights of classification loss, fusion consistency loss, and modality difference loss, respectively; the loss weights for each modality. ,in These are the heteroscedasticity noise parameters for each mode; The weights of each loss term in the joint loss function are dynamically adjusted based on randomness and uncertainty. Random uncertainty also serves as a priori basis for gradient direction control, based on heteroscedasticity noise parameters. ,by The gradient direction corresponding to the smallest mode is used as the gradient direction of the global fusion feature. A reliable reference direction.

6. The deep learning-based multimodal signal recognition method according to claim 5, characterized in that, In step S6, the directional consistency coefficient is calculated and used for gradient control in the following way: In each backpropagation iteration, the modal gradient direction generated by each modal feature extraction subnetwork is calculated. With global fusion feature gradient direction Cosine similarity between ; Calculating the directional consistency coefficient based on cosine similarity : ； in, For linear rectified activation function, when When <0, =0, indicating that the modal gradient conflicts with the global optimization objective in terms of direction; The directional consistency coefficient With modal weights Multiplying them together yields the integrated gradient weights for each mode. ; Calculate the integrated gradient : ； To integrate gradients Update all optimizable parameters of the multi-branch feature extraction network, the cross-modal latent query cross-attention fusion network, and the classification and recognition network.

7. The multimodal signal recognition method based on deep learning according to claim 1, characterized in that, In step S7, the preset convergence condition is: At the end of each training cycle, calculate the orientation consistency coefficients of each mode obtained in step S6 across all iterations within the current cycle. The variance when continuous The iteration terminates when the variance within a training cycle is less than a preset variance threshold, the decrease in the joint loss function value obtained in step S5 is less than a preset loss threshold, and the cognitive uncertainty obtained in step S5 is less than a preset confidence threshold.

8. The multimodal signal recognition method based on deep learning according to claim 7, characterized in that, In step S7, the preset probability ranges from 0.1 to 0.5, and a linear decay strategy is adopted during training to gradually decay from the initial value to the final value. Simultaneously, a knowledge distillation strategy is adopted, using the network output under complete modal input as a soft label to guide the network output under modal missing conditions to approach the soft label. The distillation loss adopts the KL divergence loss function.

9. A multimodal signal recognition device based on deep learning, characterized in that, include: The signal acquisition and preprocessing module is used to acquire multiple modal raw signals in the scene to be identified, and to preprocess and align the original signals of each modality in time and space to obtain standardized multimodal signals. The multi-branch feature extraction module is used to construct a multi-branch feature extraction network based on standardized multimodal signals and extract deep feature maps of each modality signal; The cross-modal feature fusion module is used to fuse deep feature maps into a global fusion feature vector through an attention mechanism; The classification and recognition module is used to output recognition results through a classification and recognition network based on the global fusion feature vector. The joint loss construction module is used to construct a joint loss function based on the classification and recognition task, and the weights of each loss term in the joint loss function are dynamically adjusted. The gradient control optimization module is used for backpropagation optimization based on the joint loss function. During the backpropagation optimization process, the directional consistency coefficient is calculated, and the gradient update direction and update magnitude of each modality parameter are adjusted based on the directional consistency coefficient to update the network parameters. The modality discarding and iterative training module is used to randomly discard the complete data channels of some modalities in the standardized multimodal signal with a preset probability, and repeatedly execute the classification and recognition, joint loss construction and gradient control optimization process until the preset convergence condition is met and the iteration is terminated.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the deep learning-based multimodal signal recognition method according to any one of claims 1 to 8.

Images