Multi-modal data fusion method and device, equipment and storage medium

Abstract

The invention discloses a multi-modal data fusion method and device, equipment and a storage medium, and relates to the field of data processing, and the method comprises the steps: obtaining all modal original data collected by all modal sensors, and carrying out the feature extraction of all the modal original data; determining an evaluation result based on the extraction result corresponding to each modal original data and a preset data quality evaluation rule, and determining a target weight by using the evaluation result, the current historical task result, each extraction result, a preset weight constraint condition and a preset weight adjustment rule; and based on a preset comparative learning loss function, each extraction result and the corresponding target weight, performing fusion to determine a feature fusion result, so that the feature fusion result is used for executing an adaptive target downstream task. According to the method, a closed-loop dynamic weight updating mechanism can be effectively realized, and the fusion efficiency, reliability and decision accuracy of the multi-modal data in a dynamic environment, a cross-task scene and resource-constrained equipment are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multimodal data fusion method, device, equipment and storage medium. Background Art

[0002] Multimodal data fusion is widely used in scenarios such as autonomous driving, smart healthcare, virtual reality, and smart security. However, how to efficiently handle the dynamic correlation and environmental adaptability between modalities remains a core challenge in current research.

[0003] Existing related solutions mainly perform static data fusion by setting fixed weights or eliminating low-contribution modes through offline analysis (such as retaining only sensor data with a high signal-to-noise ratio). Not only are they unable to respond to real-time environmental changes in the fused data and adaptively switch modal priorities, resulting in insufficient environmental adaptability, but when a single modality fails (such as a speech recognition failure), the static weight cannot automatically suppress the impact of the failed mode, and the resulting modal imbalance will lead to decision-making risks. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a multimodal data fusion method, apparatus, device, and storage medium that can address the problems of insufficient environmental adaptability and modal imbalance caused by fixed weights in existing solutions, effectively implement a closed-loop dynamic weight update mechanism, and improve the efficiency, reliability, and decision-making accuracy of multimodal data fusion in dynamic environments, cross-task scenarios, and resource-constrained devices. The specific solution is as follows:

[0005] In a first aspect, the present application provides a multimodal data fusion method, comprising:

[0006] Acquire each modal raw data collected by each modal sensor, and perform feature extraction on each modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each modal raw data;

[0007] Determine an evaluation result based on each of the extraction results and a preset data quality evaluation rule, and use the evaluation result, the current historical task result, each of the extraction results, the preset weight constraint condition and the preset weight adjustment rule to determine the current target weight corresponding to each of the extraction results;

[0008] Based on the preset feature fusion strategy, the preset contrast learning loss function, each of the extraction results and the corresponding target weights are fused to determine the feature fusion result, so that the feature fusion result can be used to perform the adapted target downstream task.

[0009] Optionally, the performing feature extraction on each of the modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each of the modal raw data includes:

[0010] Extracting basic features, target detection features, and semantic segmentation features from the original image data and / or original video data based on the first feature extraction model to determine a first extraction result;

[0011] Extracting acoustic features and semantic features from the original speech data based on the second feature extraction model to determine a second extraction result;

[0012] Extracting contextual semantic features and entity relationship features from the original text data based on the third feature extraction model to determine a third extraction result;

[0013] Feature extraction is performed on the original point cloud data based on the fourth feature extraction model, and frequency domain conversion is performed on the original inertial data corresponding to the inertial measurement unit to determine a fourth extraction result; the fourth extraction result includes point cloud features and frequency domain features corresponding to the inertial measurement unit.

[0014] Optionally, before performing feature extraction on each of the modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each of the modal raw data, the method further includes:

[0015] resizing the original image data and / or the original video data based on a preset size, and normalizing the corresponding resized data to determine first processed data;

[0016] Performing frame processing on the original speech data, and generating a mel-spectrogram based on the corresponding framed data to determine second processed data;

[0017] Performing word segmentation processing on the original text data based on a preset word embedding algorithm, and performing word embedding using the corresponding word segmentation results to complete a word vector sequence conversion operation, and obtaining third processed data when it is determined based on a preset sequence length that a sequence truncation operation or a sequence padding operation is not currently triggered;

[0018] downsampling the original point cloud data, and normalizing the corresponding downsampled data to determine fourth processed data;

[0019] The raw inertial data is bandpass filtered to determine fifth processed data.

[0020] Optionally, determining the evaluation result based on each of the extraction results and preset data quality evaluation rules includes:

[0021] Convolving the first extraction result based on a first preset data quality assessment rule, and calculating a response value variance using the corresponding convolution result to determine a target clarity value;

[0022] performing voice activity detection on the second extraction result based on a second preset data quality assessment rule, and performing energy comparison between the speech segment and the silence segment according to the corresponding detection result to determine a target speech signal-to-noise ratio;

[0023] Performing a data integrity check based on the fourth extraction result, and determining a target density value according to the corresponding check result;

[0024] The target clarity value, the target speech signal-to-noise ratio, and the target density value are normalized to determine an evaluation result.

[0025] Optionally, the use of the evaluation result, the current historical task result, each of the extraction results, the preset weight constraint condition and the preset weight adjustment rule to determine the current target weight corresponding to each of the extraction results includes:

[0026] Splicing the extraction results to determine a splicing result;

[0027] Determining a similarity score between each modality based on a multi-head attention mechanism and the splicing result, and determining an initial weight vector according to the similarity score;

[0028] Generate a current reward signal based on a preset reward function, the evaluation result, and historical task results; the historical task results include a satisfaction index, a first performance index corresponding to the historical classification task, and a second performance index corresponding to the historical detection task;

[0029] Performing weight updates based on the reward signal, the preset reinforcement learning algorithm framework, the initial weight vector, the preset weight constraint, and each of the extraction results to determine a current target weight corresponding to each of the extraction results;

[0030] The reinforcement learning algorithm framework includes a policy network and a value network, and the preset weight constraint conditions include weight threshold information and weight update frequency limit information corresponding to each modality.

[0031] Optionally, determining the similarity score between the modalities based on the multi-head attention mechanism and the splicing result includes:

[0032] The attention weight parameters are compressed based on a low-rank matrix decomposition algorithm, and the similarity score is determined using the corresponding compressed parameters, the multi-head attention mechanism, and the splicing result.

[0033] Optionally, the fusing based on a preset feature fusion strategy, a preset contrastive learning loss function, each of the extraction results and the corresponding target weight includes:

[0034] Performing feature dimension alignment based on a preset dimension alignment rule, a preset contrastive learning loss function, and each of the extraction results to determine a dimension alignment result;

[0035] Performing weighted fusion of features based on a preset weight broadcast mechanism, the dimension alignment result, and the target weight to determine fused features;

[0036] The corresponding fused features are enhanced based on the self-attention mechanism, and channel paper cutting is performed on the corresponding enhanced features to determine the feature fusion results.

[0037] In a second aspect, the present application provides a multimodal data fusion device, comprising:

[0038] A feature extraction module is used to obtain each modal raw data collected by each modal sensor, and perform feature extraction on each modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each modal raw data;

[0039] A weight determination module is used to determine an evaluation result based on each of the extraction results and preset data quality evaluation rules, and to determine a target weight currently corresponding to each of the extraction results using the evaluation result, the current historical task result, each of the extraction results, preset weight constraints, and preset weight adjustment rules;

[0040] A fusion result determination module is used to fuse the features based on a preset feature fusion strategy, a preset contrast learning loss function, each of the extraction results and the corresponding target weights to determine the feature fusion result, so that the feature fusion result can be used to perform the adapted target downstream task.

[0041] In a third aspect, the present application provides an electronic device, comprising:

[0042] Memory, used to store computer programs;

[0043] The processor is configured to execute the computer program to implement the steps of the aforementioned multimodal data fusion method.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which implements the steps of the aforementioned multimodal data fusion method when executed by a processor.

[0045] It can be seen that in this application, each modal raw data collected by each modal sensor is obtained, and feature extraction is performed on each modal raw data based on a preset feature extraction rule to determine the extraction results corresponding to each modal raw data; the evaluation results are determined based on each extraction result and the preset data quality assessment rule, and the target weight corresponding to each extraction result is determined by using the evaluation results, the current historical task results, each extraction result, the preset weight constraint and the preset weight adjustment rule; based on the preset feature fusion strategy, the preset contrast learning loss function, each extraction result and the corresponding target weight are fused to determine the feature fusion result, so that the feature fusion result can be used to perform the adapted target downstream task. That is to say, in this application, the corresponding feature extraction rules are first applied to extract features from each modal raw data, and then data quality assessment is performed based on the feature set removal result, and then the weight is dynamically adjusted using the evaluation results, the current historical task results, the extraction results, and the preset weight constraint to determine the target weight, and then the extraction results are fused based on the target weight and the preset contrast learning loss function, so that the determined feature fusion result can be used to perform the adapted target downstream task. In this way, the problems of insufficient environmental adaptability and modal imbalance caused by fixed weights in existing solutions can be solved, a closed-loop dynamic weight update mechanism can be effectively implemented, and the efficiency, reliability and decision-making accuracy of the fusion of multimodal data in dynamic environments, cross-task scenarios, and resource-constrained devices can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 A flow chart of a multimodal data fusion method provided in this application;

[0048] Figure 2 A schematic diagram of the structure of a multimodal data fusion device provided in this application;

[0049] Figure 3 This is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] Existing related solutions mainly perform static data fusion, by setting fixed weights, or eliminating low-contribution modes through offline analysis (such as retaining only sensor data with a high signal-to-noise ratio). Not only can they not cope with real-time environmental changes to fuse data, they cannot adaptively switch modal priorities, resulting in insufficient environmental adaptability. Moreover, when a single modality fails (such as speech recognition failure), the static weights cannot automatically suppress the impact of the failed mode, and the resulting modal imbalance will cause decision-making risks. To this end, the present application provides a multimodal data fusion solution that can solve the problems of insufficient environmental adaptability and modal imbalance caused by fixed weights in existing solutions, effectively implement a closed-loop dynamic weight update mechanism, and improve the efficiency, reliability, and decision-making accuracy of multimodal data fusion in dynamic environments, cross-task scenarios, and resource-constrained devices.

[0052] See also Figure 1 As shown, an embodiment of the present invention discloses a multimodal data fusion method, including:

[0053] Step S11: acquiring each modal raw data collected by each modal sensor, and performing feature extraction on each modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each modal raw data.

[0054] In this embodiment, during the data acquisition phase, various modal sensors (such as cameras, microphones, and lidar) collect data, and then extract data features. The data includes image / video data, voice / audio data, text data, and sensor data (this embodiment uses point cloud data and inertial data as examples to illustrate the subsequent process). Specifically, a first feature extraction model is used to extract basic features, object detection features, and semantic segmentation features from the original image data and / or original video data to determine a first extraction result. A second feature extraction model is used to extract acoustic and semantic features from the original voice data to determine a second extraction result. A third feature extraction model is used to extract contextual semantic features and entity relationship features from the original text data to determine a third extraction result. A fourth feature extraction model is used to extract features from the original point cloud data, and the original inertial data corresponding to the inertial measurement unit is converted to the frequency domain to determine a fourth extraction result. The fourth extraction result includes point cloud features and frequency domain features corresponding to the inertial measurement unit. Furthermore, for video data, optical flow can be used to extract motion features from adjacent frames.

[0055] Specifically, in the feature extraction stage, for image / video data, basic features: based on the convolutional neural network convolution backbone, output 7×7×2048-dimensional feature map, and obtain 2048-dimensional vector after global average pooling; target detection features: YOLOv5s target detection model, output target bounding box coordinates (x, y, w, h) and category confidence (N×6-dimensional vector, N is the number of detected targets); semantic segmentation features: DeepLabv3+ model, output H / 16×W / 16×21-dimensional semantic mask, model output format: Fimage=[ResNet feature, YOLO target vector, DeepLab mask]; for speech data, acoustic features: CNN architecture (Convolutional Neural The model uses a convolutional neural network (CNN) to input a mel-spectrogram and output a 128-dimensional acoustic feature vector. Semantic features are implemented using a Transformer encoder, which takes the original audio waveform as input and outputs a 768-dimensional semantic vector (including keywords, sentiment polarity, and other information). The model output format is: Faudio = [VGGish acoustic features, Wav2Vec2.0 semantic features]. For text data, contextual semantic features are implemented using a BERT-base model (Bidirectional Encoder Representations from Transformers-base, a pre-trained language model) that outputs a 512×768-dimensional token embedding. This is then pooled with CLStoken (Classification token) to obtain a 768-dimensional sentence vector. Entity relationship features are implemented using an LSTM+CRF architecture (Long Short-Term Memory+Conditional Random Access Memory). Field, long short-term memory network + conditional random field), input BERT hidden layer output, identify named entities and dependencies, and output N×(entity type + relationship type) dimensional vectors (N is the number of entities). The model output format is: Ftext=[BERT sentence embedding, LSTM entity relationship vector]; for point cloud data, the PointNet++ model is used to output a 1024-dimensional global feature vector; for inertial data, a Fourier transform is performed on the data of each axis, and the first 50 low-frequency components (corresponding to 0-5Hz frequency) are taken to generate 6×50 dimensional frequency domain features. The output format is: Fsensor=[PointNet++ point cloud features, inertial measurement unit frequency domain features].

[0056] It can be understood that before feature extraction, the collected data is also preprocessed, that is, the original image data and / or the original video data are resized based on a preset size, and the corresponding adjusted data are normalized to determine the first processed data; the original voice data is framed, and a Mel-spectrogram is generated based on the corresponding framed data to determine the second processed data; the original text data is segmented based on a preset word embedding algorithm, and word embedding is performed using the corresponding segmentation results to complete the word vector sequence conversion operation, and when it is determined based on the preset sequence length that the sequence truncation operation or sequence filling operation is not currently triggered, the third processed data is obtained; the original point cloud data is downsampled, and the corresponding downsampled data is normalized to determine the fourth processed data; the original inertial data is bandpass filtered to determine the fifth processed data.

[0057] Specifically, in the preprocessing stage, for image / video data, resize to a fixed size (such as 224×224), and normalize pixel values ​​to [0,1]. For speech data, frame processing is performed: frame length is 50ms, frame shift is 25ms, Hamming window is added, and Mel spectrum is calculated: 80 Mel filters, FFT points are 512, and a T×80-dimensional spectrum is obtained. For text data, word embedding is performed: BERT Tokenizer is used for word segmentation, input ID, segment ID, attention mask are generated, and sequence truncation / padding is performed to fix the sequence length to 512. For point cloud data, downsample to 10k points and normalize the coordinates to [-1,1]. For inertial data, DC offset is removed and 0.1-50Hz bandpass filtering is performed.

[0058] In addition, if missing features are found, the missing modal features can be supplemented based on historical data through the generative model.

[0059] Step S12: Determine the evaluation results based on each of the extraction results and the preset data quality evaluation rules, and use the evaluation results, the current historical task results, each of the extraction results, the preset weight constraints and the preset weight adjustment rules to determine the current target weight corresponding to each of the extraction results.

[0060] In this embodiment, after completing feature extraction, before calculating the weight, it is necessary to evaluate the quality of the collected data to analyze its reliability, that is, convolve the first extraction result based on the first preset data quality evaluation rule, and use the corresponding convolution result to calculate the response value variance to determine the target clarity value; perform voice activity detection on the second extraction result based on the second preset data quality evaluation rule, and perform energy comparison between the voice segment and the silence segment based on the corresponding detection result to determine the target voice signal-to-noise ratio; perform data integrity check based on the fourth extraction result, and determine the target density value based on the corresponding check result; normalize the target clarity value, the target voice signal-to-noise ratio and the target density value to determine the evaluation result.

[0061] Specifically, during the quality assessment phase, the target clarity value for image data can be determined based on the Laplacian variance (VOL). The image is first converted to grayscale, convolved with a 3×3 Laplacian kernel, and the variance of the response is calculated. For speech data, the target signal-to-noise ratio (SNR) can be determined based on the ratio of speech energy to noise energy using short-term energy detection (VAD). The energy of the silence segment is used as a noise estimate, and the SNR is determined based on the ratio of speech energy to noise energy. Furthermore, a smoothing filter (such as an exponential moving average with α = 0.9) can be used to reduce SNR fluctuations. For sensor data, the target density value can be determined by examining the density of the point cloud and the missing rate of the inertial measurement unit (IMU). Subsequently, using a cross-modal normalization strategy, the target clarity value, target SNR, and target density value are normalized to the range [0, 1]. The corresponding reliability score, i.e., the assessment result, is then output.

[0062] After completing the data quality assessment, the target weight is determined using the assessment results, the current historical task results and the preset weight adjustment rules, that is, the extraction results are spliced ​​to determine the splicing result; the similarity scores between the modalities are determined based on the multi-head attention mechanism and the splicing results, and the initial weight vector is determined based on the similarity score; the current reward signal is generated based on the preset reward function, the assessment results and the historical task results; the historical task results include a satisfaction index, a first performance index corresponding to the historical classification task (for example, classification accuracy, recall rate, etc.) and a second performance index corresponding to the historical detection task (for example, detection delay, missed detection rate, etc.); the weight is updated based on the reward signal, the preset reinforcement learning algorithm framework, the initial weight vector, the preset weight constraint conditions and the extraction results to determine the current target weight corresponding to each extraction result; wherein the reinforcement learning algorithm framework includes a policy network and a value network, and the preset weight constraint conditions include weight threshold information and weight update frequency limit information corresponding to each modality. Among them, the similarity score between each modality is determined based on the multi-head attention mechanism and the splicing result, including: compressing the attention weight parameters based on the low-rank matrix decomposition algorithm, and using the corresponding compressed parameters, the multi-head attention mechanism and the splicing result to determine the similarity score.

[0063] Specifically, in the weight determination stage, based on the cross-modal attention mechanism, taking the first time as an example, it is necessary to calculate the initial weight, input the feature vectors of each modality into the multi-head attention mechanism, calculate the similarity score of Query (target modality) and Key (other modalities), support hierarchical attention (such as calculating the image-speech association first, and then the secondary association with the text), and generate the initial weight vector based on the similarity score. =[w1,w2,…,wn], and the sum of the initial weights of all modes (a total of n modal data) must be 1, that is, = 1. If the data of a certain modality is missing in this part, based on the dynamic weight compensation strategy, the initial weight of the modality is forced to 0, and the initial weights of other modalities are normalized.

[0064] Afterwards, dynamic optimization of reinforcement learning is performed based on the initial weights. Based on the preset reward function, multi-objective optimization is achieved by comprehensively considering task accuracy, data reliability, and computational cost. The reward function is:

[0065] .

[0066] Where, Represents the task accuracy indicator, which is determined based on the current historical task results; Represents the data reliability index, i.e., the evaluation result determined in the above steps; represents the computational cost indicator; is the weight coefficient, which can be customized to a specific value or the system default configuration. Then, using the strategy network-value network architecture, the strategy network determines the dynamic weight based on the output of the reward function. , the value network evaluates the state value and updates the policy parameters through gradient descent . Then the target weight is determined by combining the initial weight and dynamic weight.

[0067] Furthermore, regarding the weight constraints, a single-mode minimum weight threshold is set to avoid complete mode failure, and exponential smoothing filtering is performed when necessary ( = 0.9) to suppress fluctuations; limit the weight update frequency, for example, no more than 10 updates per second to adapt to real-time scenarios.

[0068] At the same time, regarding the corresponding lightweight design, low-rank matrix decomposition is used to reduce the complexity of attention weight calculation, and a weight caching mechanism is configured. For short-term stable scenarios, historical weights can be reused to reduce repeated calculations.

[0069] Step S13: Based on the preset feature fusion strategy, the preset contrastive learning loss function, each of the extraction results and the corresponding target weights are fused to determine a feature fusion result, so that the feature fusion result can be used to perform the adapted target downstream task.

[0070] In this embodiment, after determining the target weight, feature fusion is triggered. Specifically, feature dimension alignment is performed based on a preset dimension alignment rule, a preset contrastive learning loss function, and the extraction results to determine the dimension alignment result. A weighted fusion of features is performed based on a preset weight broadcast mechanism, the dimension alignment result, and the target weight to determine the fused feature. Feature enhancement is performed on the corresponding fused features based on a self-attention mechanism, and channel paper cutting is performed on the corresponding enhanced features to determine the feature fusion result. It is understood that since the dimensions of features from different modalities may be inconsistent, dimensionality must be unified to support weighted summation. Therefore, this embodiment performs feature dimension alignment. Specifically, a modality-specific mapping layer is performed to map the features of each modality to a unified dimension through the application of a fully connected layer or a convolutional layer. The aligned features are then fused. The vector representation corresponding to the target weight is expanded to the corresponding dimension based on a preset weight broadcast mechanism to fit the feature matrix composed of the aligned features. This allows for accurate multiplication of the features and weights of each modality to complete feature-level fusion. Furthermore, after determining the target weight, decision-level feature fusion can also be performed based on the target weight, such as for joint decision-making based on imaging and medical records in medical diagnosis. To improve the consistency of the semantic space, this embodiment applies a preset contrastive learning loss function to achieve effective cross-modal semantic alignment by maximizing the similarity between related modal features while minimizing the similarity between unrelated modal features. By minimizing this loss function, the model learns to bring the feature vectors of related modalities closer together and push the feature vectors of unrelated modalities further apart. For example, in an intelligent customer service system, the contrastive learning loss function can be used to more closely align the acoustic features of speech, the semantic features of chat text, and the visual features of camera expressions in the semantic space, thereby improving the accuracy of intent recognition.

[0071] Furthermore, after obtaining the fused features, feature enhancement is performed. Layer normalization can be used to alleviate the vanishing gradient problem caused by differences in numerical distribution between modalities, and / or a self-attention mechanism can be applied to the fused features to further capture cross-modal interaction information. Furthermore, to improve efficiency, mixed-precision computation can be performed, and / or channel pruning can be performed on the fused features to reduce the computational effort of subsequent tasks, ultimately yielding the final feature fusion result.

[0072] After determining the feature fusion result, the result will be matched with downstream tasks, and the adapted task will be selected for execution. Downstream tasks include classification tasks (such as intent recognition), detection tasks (obstacle category detection), regression tasks (such as obstacle distance prediction and user emotion rating), and generation tasks (such as text generation and image generation and speech synthesis).

[0073] In addition, in order to further reduce the consumption of computing resources, this embodiment has developed relevant lightweight designs: (1) low-rank matrix decomposition. Taking the intelligent customer service system as an example, when processing a large amount of user voice, chat text and camera expression data, low-rank matrix decomposition can significantly reduce the amount of calculation while ensuring the fusion effect, thereby improving the real-time response capability of the system; (2) weight caching mechanism. When it is detected that certain characteristics of the scene (such as time, user behavior pattern, etc.) have not changed significantly, the historical weights are directly reused. For example, in an autonomous driving perception system, when a vehicle is driving on a road with relatively stable road conditions, the data characteristics of the camera, lidar and millimeter-wave radar will not change much in a short period of time. At this time, the weight caching mechanism can avoid repeated calculation of weights, save computing resources, and thus improve the real-time performance of the system.

[0074] At the same time, this embodiment also designs a corresponding edge computing optimization strategy. When fusing multimodal data on end-side devices (such as vehicle-mounted chips, smart terminals, etc.), in order to reduce the original data transmission bandwidth, during the data collection phase, each modal sensor (such as a camera, microphone, lidar, etc.) transmits the collected data to the edge node, and processes the data on the edge node until the dynamic weight is determined. This can significantly reduce the amount of data transmission. For example, in autonomous driving scenarios, this method can save a lot of sensor data traffic, thereby alleviating network bandwidth pressure and improving the real-time performance and stability of the system.

[0075] In summary, this embodiment provides a multimodal data fusion solution based on dynamic modal weights. The present invention constructs a dynamic weight adjustment framework through the collaborative optimization of cross-modal attention mechanism and reinforcement learning, and jointly optimizes dynamic weights and downstream tasks based on an end-to-end training strategy. The dynamic weight generation module is connected to the downstream task head, and end-to-end training is performed through back-propagation. In this way, a closed-loop dynamic weight mechanism of environmental perception-weight adjustment-fusion optimization is constructed, which improves the robustness in complex scenarios, reduces the computing power consumption of end-side devices, and suppresses the interference of low-quality modalities in real time. It can significantly improve the fusion efficiency and decision-making accuracy of multimodal data in dynamic environments, cross-task scenarios, and resource-constrained devices.

[0076] It can be seen that in this application, the corresponding feature extraction rules are first applied to extract features from the original data of each modality, and then the data quality is evaluated based on the result of the feature set. Then, the weight is dynamically adjusted using the evaluation results, the current historical task results, the extraction results, and the preset weight constraints to determine the target weight. The extraction results are then fused based on the target weight and the preset contrast learning loss function so that the determined feature fusion results can be used to perform the adapted target downstream tasks. In this way, the problems of insufficient environmental adaptability and modal imbalance caused by fixed weights in the existing scheme can be solved, and a closed-loop dynamic weight update mechanism can be effectively realized, thereby improving the efficiency, reliability, and decision-making accuracy of the fusion of multimodal data in dynamic environments, cross-task scenarios, and resource-constrained devices.

[0077] See also Figure 2 As shown, the embodiment of the present application also discloses a multimodal data fusion device, including:

[0078] The feature extraction module 11 is used to obtain each modal raw data collected by each modal sensor, and perform feature extraction on each modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each modal raw data;

[0079] A weight determination module 12 is configured to determine an evaluation result based on each of the extraction results and a preset data quality evaluation rule, and to determine a target weight currently corresponding to each of the extraction results using the evaluation result, the current historical task result, each of the extraction results, the preset weight constraint condition, and the preset weight adjustment rule;

[0080] The fusion result determination module 13 is used to fuse the feature fusion results based on the preset feature fusion strategy, the preset contrast learning loss function, the extraction results and the corresponding target weights to determine the feature fusion result, so that the feature fusion result can be used to perform the adapted target downstream task.

[0081] It can be seen that in this application, the corresponding feature extraction rules are first applied to extract features from the original data of each modality, and then the data quality is evaluated based on the result of the feature set. Then, the weight is dynamically adjusted using the evaluation results, the current historical task results, the extraction results, and the preset weight constraints to determine the target weight. The extraction results are then fused based on the target weight and the preset contrast learning loss function so that the determined feature fusion results can be used to perform the adapted target downstream tasks. In this way, the problems of insufficient environmental adaptability and modal imbalance caused by fixed weights in the existing scheme can be solved, and a closed-loop dynamic weight update mechanism can be effectively realized, thereby improving the efficiency, reliability, and decision-making accuracy of the fusion of multimodal data in dynamic environments, cross-task scenarios, and resource-constrained devices.

[0082] In some specific embodiments, the feature extraction module 11 may specifically include:

[0083] A first extraction unit is configured to extract basic features, object detection features, and semantic segmentation features from the original image data and / or original video data based on a first feature extraction model to determine a first extraction result;

[0084] A second extraction unit is used to extract acoustic features and semantic features from the original speech data based on a second feature extraction model to determine a second extraction result;

[0085] A third extraction unit is used to extract contextual semantic features and entity relationship features from the original text data based on a third feature extraction model to determine a third extraction result;

[0086] A fourth extraction unit is used to extract features from the original point cloud data based on a fourth feature extraction model, and to perform frequency domain conversion on the original inertial data corresponding to the inertial measurement unit to determine a fourth extraction result; the fourth extraction result includes point cloud features and frequency domain features corresponding to the inertial measurement unit.

[0087] In some specific embodiments, the multimodal data fusion device may further include:

[0088] a first processing unit, configured to resize the original image data and / or the original video data based on a preset size, and normalize the corresponding resized data to determine first processed data;

[0089] A second processing unit is configured to perform frame processing on the original speech data and generate a Mel-spectrogram based on the corresponding framed data to determine second processed data;

[0090] a third processing unit, configured to perform word segmentation processing on the original text data based on a preset word embedding algorithm, and perform word embedding using the corresponding word segmentation results to complete a word vector sequence conversion operation, and obtain third processed data when it is determined based on a preset sequence length that a sequence truncation operation or a sequence padding operation is not currently triggered;

[0091] a fourth processing unit, configured to downsample the original point cloud data and normalize the corresponding downsampled data to determine fourth processed data;

[0092] The fifth processing unit is configured to perform bandpass filtering on the raw inertial data to determine fifth processed data.

[0093] In some specific embodiments, the weight determination module 12 may specifically include:

[0094] a clarity value determining unit, configured to convolve the first extraction result based on a first preset data quality assessment rule, and calculate a response value variance using the corresponding convolution result to determine a target clarity value;

[0095] a signal-to-noise ratio determination unit, configured to perform voice activity detection on the second extraction result based on a second preset data quality assessment rule, and perform energy comparison between the speech segment and the silence segment according to the corresponding detection result to determine a target speech signal-to-noise ratio;

[0096] a density value determining unit, configured to perform a data integrity check based on the fourth extraction result, and determine a target density value according to a corresponding check result;

[0097] An evaluation result determination unit is used to normalize the target clarity value, the target speech signal-to-noise ratio and the target density value to determine an evaluation result.

[0098] In some specific embodiments, the weight determination module 12 may specifically include:

[0099] A splicing unit, configured to splice the extraction results to determine a splicing result;

[0100] an initial weight determination unit, configured to determine a similarity score between each modality based on a multi-head attention mechanism and the splicing result, and determine an initial weight vector according to the similarity score;

[0101] a performance indicator determination unit, configured to generate a current reward signal based on a preset reward function, the evaluation result, and historical task results; the historical task results including a satisfaction indicator, a first performance indicator corresponding to the historical classification task, and a second performance indicator corresponding to the historical detection task;

[0102] A weight update unit is used to update the weight based on the reward signal, the preset reinforcement learning algorithm framework, the initial weight vector, the preset weight constraint conditions and each of the extraction results to determine the current target weight corresponding to each of the extraction results; wherein, the reinforcement learning algorithm framework includes a policy network and a value network, and the preset weight constraint conditions include weight threshold information and weight update frequency limit information corresponding to each modality.

[0103] In some specific embodiments, the initial weight determination unit may specifically include:

[0104] The similarity calculation subunit is used to compress the attention weight parameters based on the low-rank matrix decomposition algorithm, and determine the similarity score using the corresponding compressed parameters, the multi-head attention mechanism and the splicing result.

[0105] In some specific embodiments, the multimodal data fusion device may further include:

[0106] A dimension alignment unit, configured to perform feature dimension alignment based on a preset dimension alignment rule, a preset contrastive learning loss function, and each of the extraction results to determine a dimension alignment result;

[0107] A weighted fusion unit, configured to perform weighted fusion of features based on a preset weight broadcast mechanism, the dimension alignment result, and the target weight to determine a fused feature;

[0108] The feature enhancement unit is used to enhance the corresponding fused features based on the self-attention mechanism and perform channel paper cutting on the corresponding enhanced features to determine the feature fusion result.

[0109] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0110] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the multimodal data fusion method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0111] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0112] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0113] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the multimodal data fusion method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0114] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the multimodal data fusion method disclosed above is implemented. For the specific steps of this method, please refer to the corresponding content disclosed in the above embodiments and will not be repeated here.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0116] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0118] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0119] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multimodal data fusion method, characterized in that: include: Acquire each modal raw data collected by each modal sensor, and perform feature extraction on each modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each modal raw data; Determine an evaluation result based on each of the extraction results and a preset data quality evaluation rule, and use the evaluation result, the current historical task result, each of the extraction results, the preset weight constraint condition and the preset weight adjustment rule to determine the current target weight corresponding to each of the extraction results; Based on the preset feature fusion strategy, the preset contrast learning loss function, each of the extraction results and the corresponding target weights are fused to determine the feature fusion result, so that the feature fusion result can be used to perform the adapted target downstream task.

2. The multimodal data fusion method according to claim 1, characterized in that: The performing feature extraction on each of the modal original data based on a preset feature extraction rule to determine an extraction result corresponding to each of the modal original data includes: Extracting basic features, target detection features, and semantic segmentation features from the original image data and / or original video data based on the first feature extraction model to determine a first extraction result; Extracting acoustic features and semantic features from the original speech data based on the second feature extraction model to determine a second extraction result; Extracting contextual semantic features and entity relationship features from the original text data based on the third feature extraction model to determine a third extraction result; Feature extraction is performed on the original point cloud data based on the fourth feature extraction model, and frequency domain conversion is performed on the original inertial data corresponding to the inertial measurement unit to determine a fourth extraction result; the fourth extraction result includes point cloud features and frequency domain features corresponding to the inertial measurement unit.

3. The multimodal data fusion method according to claim 2, characterized in that: Before performing feature extraction on each of the modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each of the modal raw data, the method further includes: resizing the original image data and / or the original video data based on a preset size, and normalizing the corresponding resized data to determine first processed data; Performing frame processing on the original speech data, and generating a mel-spectrogram based on the corresponding framed data to determine second processed data; Performing word segmentation processing on the original text data based on a preset word embedding algorithm, and performing word embedding using the corresponding word segmentation results to complete a word vector sequence conversion operation, and obtaining third processed data when it is determined based on a preset sequence length that a sequence truncation operation or a sequence padding operation is not currently triggered; downsampling the original point cloud data, and normalizing the corresponding downsampled data to determine fourth processed data; The raw inertial data is bandpass filtered to determine fifth processed data.

4. The multimodal data fusion method according to claim 2, characterized in that: Determining the evaluation result based on each of the extraction results and the preset data quality evaluation rules includes: Convolving the first extraction result based on a first preset data quality assessment rule, and calculating a response value variance using the corresponding convolution result to determine a target clarity value; performing voice activity detection on the second extraction result based on a second preset data quality assessment rule, and performing energy comparison between the speech segment and the silence segment according to the corresponding detection result to determine a target speech signal-to-noise ratio; Performing a data integrity check based on the fourth extraction result, and determining a target density value according to the corresponding check result; The target clarity value, the target speech signal-to-noise ratio, and the target density value are normalized to determine an evaluation result.

5. The multimodal data fusion method according to claim 1, characterized in that: The method of using the evaluation result, the current historical task result, each of the extraction results, the preset weight constraint condition and the preset weight adjustment rule to determine the current target weight corresponding to each of the extraction results includes: Splicing the extraction results to determine a splicing result; Determining a similarity score between each modality based on a multi-head attention mechanism and the splicing result, and determining an initial weight vector according to the similarity score; Generate a current reward signal based on a preset reward function, the evaluation result, and historical task results; the historical task results include a satisfaction index, a first performance index corresponding to the historical classification task, and a second performance index corresponding to the historical detection task; Performing weight updates based on the reward signal, the preset reinforcement learning algorithm framework, the initial weight vector, the preset weight constraint, and each of the extraction results to determine a current target weight corresponding to each of the extraction results; The reinforcement learning algorithm framework includes a policy network and a value network, and the preset weight constraint conditions include weight threshold information and weight update frequency limit information corresponding to each modality.

6. The multimodal data fusion method according to claim 5, characterized in that: The determining of the similarity scores between the modalities based on the multi-head attention mechanism and the splicing result includes: The attention weight parameters are compressed based on a low-rank matrix decomposition algorithm, and the similarity score is determined using the corresponding compressed parameters, the multi-head attention mechanism, and the splicing result.

7. The multimodal data fusion method according to any one of claims 1 to 6, characterized in that: The fusion based on the preset feature fusion strategy, the preset contrastive learning loss function, the extraction results and the corresponding target weights includes: Performing feature dimension alignment based on a preset dimension alignment rule, a preset contrastive learning loss function, and each of the extraction results to determine a dimension alignment result; Performing weighted fusion of features based on a preset weight broadcast mechanism, the dimension alignment result, and the target weight to determine fused features; The corresponding fused features are enhanced based on the self-attention mechanism, and channel paper cutting is performed on the corresponding enhanced features to determine the feature fusion results.

8. A multimodal data fusion device, characterized in that: include: A feature extraction module is used to obtain each modal raw data collected by each modal sensor, and perform feature extraction on each modal raw data based on a preset feature extraction rule to determine an extraction result corresponding to each modal raw data; A weight determination module is used to determine an evaluation result based on each of the extraction results and preset data quality evaluation rules, and to determine a target weight currently corresponding to each of the extraction results using the evaluation result, the current historical task result, each of the extraction results, preset weight constraints, and preset weight adjustment rules; A fusion result determination module is used to fuse the features based on a preset feature fusion strategy, a preset contrast learning loss function, each of the extraction results and the corresponding target weights to determine the feature fusion result, so that the feature fusion result can be used to perform the adapted target downstream task.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the multimodal data fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the multimodal data fusion method according to any one of claims 1 to 7.

Images