Multimodal data fusion analysis method, device, equipment and storage medium

Abstract

The application provides a multi-modal data fusion analysis method, device, equipment and storage medium, the application first collects multi-modal original data, and high-quality target data is obtained through cleaning and standardization preprocessing; then, different modal data is extracted by using corresponding algorithms to extract feature vectors; the core is to perform adaptive fusion analysis, the feature vectors are fused according to the dynamic quality parameters and the current data size of each modal data, and the fusion features are generated; finally, the final result is obtained through intelligent analysis model reasoning. The application overcomes the problems of fixed fusion strategy and poor data adaptability in the prior art, realizes dynamic adaptive adjustment of fusion weight and strategy, and effectively improves the analysis accuracy and calculation efficiency.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence technology, and in particular to a multimodal data fusion analysis method, apparatus, device, and storage medium. Background Technology

[0002] In today's digital age, data is experiencing explosive growth, and its data types are becoming increasingly diverse, encompassing multiple modalities such as text, images, audio, and video. Different modalities of data contain different dimensions of information, and the analysis of single-modal data often yields only partial conclusions, making it difficult to meet the needs of in-depth analysis in complex scenarios.

[0003] Currently, some multimodal data fusion and analysis technologies exist, but most suffer from the following shortcomings: First, the feature extraction and fusion process of multimodal data lacks a unified framework, and the processing logic of different modalities is independent, resulting in low fusion efficiency. Second, existing fusion algorithms have poor data adaptability, and the fusion strategy remains fixed, making it difficult to cope with data differences under different data scales and industry scenarios, leading to insufficient accuracy of analysis results. Third, in the Java ecosystem, there is a lack of a mature and efficient multimodal data fusion and analysis tool, requiring developers to spend a lot of effort writing and debugging the underlying logic, which seriously affects development efficiency.

[0004] For example, in intelligent customer service scenarios, it is necessary to simultaneously analyze multimodal data such as users' text inquiries, voice tone, and facial expressions to accurately understand their true needs. Current technologies often only analyze text, voice, and images independently, then simply overlay the results, failing to achieve deep fusion. Furthermore, when noise or missing data exists in a particular modality, there is a lack of effective dynamic adjustment mechanisms, easily leading to misunderstandings and reduced service quality. In medical diagnostic scenarios, the fusion of multimodal information such as medical images, medical records, and patient statements faces similar challenges. The reliability and information content of different modalities dynamically change with collection conditions, making fixed-weight fusion methods insufficient to guarantee the stability and accuracy of diagnostic results.

[0005] In summary, the problems existing in the current technology urgently need to be solved. Summary of the Invention

[0006] This invention provides a multimodal data fusion analysis method, apparatus, device, and storage medium to overcome the shortcomings of the prior art and achieve both high accuracy and computational efficiency.

[0007] This invention provides a multimodal data fusion and analysis method, comprising: Collect multimodal raw data, and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data; For each modality in the multimodal target data, a feature vector for that modality is extracted using a feature extraction algorithm corresponding to the modality type; Perform adaptive fusion analysis, and perform fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch to generate fusion features; The fused features are input into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results.

[0008] According to a multimodal data fusion analysis method provided by the present invention, the step of performing adaptive fusion analysis, which involves fusing the feature vectors based on the dynamic quality parameters of each modality and the size of the current batch of data to be processed, to generate fusion features, specifically includes: The confidence level of each modality is calculated based on the noise level, missing rate, and historical accuracy of each modality data. The fusion weight of each modality is dynamically determined based on the confidence level and the preset initial weight of industry type. Determine the target fusion strategy based on the size of the current batch of data to be processed; Based on the target fusion strategy and the fusion weights, cross-modal semantic alignment and fusion processing are performed on the feature vectors of each modality to obtain fused features.

[0009] According to the multimodal data fusion analysis method provided by the present invention, after the step of performing cross-modal semantic alignment and fusion processing on the feature vectors of each modality based on the target fusion strategy and the fusion weights to obtain fused features, the method further includes: Based on each modal data in the multimodal target data, the modal analysis results are obtained by using the corresponding single-modal analysis model. Calculate the degree of conflict between the modal analysis results; If the conflict level exceeds a preset threshold, the process will revert to dynamically determining the fusion weights of each modality and readjusting the fusion weights of each modality. Based on the readjusted fusion weights, cross-modal semantic alignment and fusion processing is performed again until the conflict degree is reduced to below the preset threshold.

[0010] According to a multimodal data fusion analysis method provided by the present invention, the confidence level of each modality is calculated by the following formula:

[0011] in, For each modality, Let m be the noise level of the m-th mode. Let m be the missing rate of the m-th mode. Let be the historical accuracy of the m-th mode.

[0012] According to the multimodal data fusion analysis method provided by the present invention, the step of performing cross-modal semantic alignment and fusion processing on the feature vectors of each modality based on the target fusion strategy and the fusion weights to obtain fused features specifically includes: The feature vectors of different modalities are mapped to a unified semantic space to achieve cross-modal semantic alignment; Based on the cross-modal attention mechanism, the correlation degree between different modal features is calculated, and the attention weight of each modal feature in the fusion process is determined based on the correlation degree. Based on the fusion weight and the attention weight, the feature vectors of each modality are weighted and fused to generate the fused feature.

[0013] According to a multimodal data fusion and analysis method provided by the present invention, the step of collecting multimodal raw data and cleaning and standardizing the multimodal raw data to obtain preprocessed multimodal target data specifically includes: Collect multimodal raw data from at least one data source, including databases, local files, or network interfaces; The multimodal raw data is cleaned, including at least one of data format standardization, missing value filling, and outlier removal; The cleaned data is standardized to generate the preprocessed multimodal target data.

[0014] According to a multimodal data fusion analysis method provided by the present invention, the step of inputting the fusion features into a preset intelligent analysis model for inference to obtain the multimodal fusion analysis result specifically includes: The fused features are input into a preset intelligent analysis model, which is a traditional machine learning model or a deep learning model. The intelligent analysis model is used to infer the fused features and output the probability distribution corresponding to each candidate analysis result. The final multimodal fusion analysis result is determined based on the probability distribution.

[0015] The present invention also provides a multimodal data fusion and analysis device, comprising: The data acquisition module is used to acquire multimodal raw data and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data. The feature extraction module is used to extract the feature vector of each modality from the multimodal target data using a feature extraction algorithm corresponding to the modality type. The feature fusion module is used to perform adaptive fusion analysis, which performs fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch, and generates fused features. The feature reasoning module is used to input the fused features into a preset intelligent analysis model for reasoning to obtain multimodal fusion analysis results.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal data fusion analysis method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal data fusion analysis method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal data fusion analysis method as described above.

[0019] The multimodal data fusion analysis method, apparatus, device, and storage medium provided by this invention effectively eliminate noise and inconsistencies in the original multimodal data by collecting, cleaning, and standardizing it, thus providing a high-quality data foundation for subsequent analysis. Feature vectors are extracted using feature extraction algorithms that match the data type for different modalities, ensuring that information from each modality is fully and accurately represented.

[0020] Based on this, the present invention performs adaptive fusion analysis. According to the dynamic quality parameters of each modality data and the scale of the data to be processed in the current batch, the feature vectors are fused to generate fused features. Finally, the fused features are input into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results, achieving the following beneficial effects: First, it significantly improves the accuracy of multimodal data fusion. This invention dynamically calculates the confidence level based on the actual quality of each modality's data (such as noise level, missing rate, and historical accuracy) and adjusts the fusion weights of each modality accordingly. This abandons the traditional method's coarse-grained fusion approach with fixed weights, allowing high-confidence modalities to play a greater role in the fusion process while effectively suppressing the influence of low-confidence modalities, thus significantly improving the accuracy of the analysis results. Taking intelligent customer service scenarios as an example, the accuracy of user request identification using this invention can reach 92.3%, far exceeding the 81.2% of traditional fixed fusion schemes.

[0021] Secondly, it greatly enhances the adaptability and versatility of the fusion strategy. This invention automatically adapts different fusion strategies based on the scale of the data to be processed in the current batch—using computationally efficient weighted fusion for small data volumes, feature concatenation fusion for medium data volumes, and deep learning fusion for large-scale data with high accuracy requirements—ensuring a high degree of matching between the fusion strategy and data features. This guarantees analytical accuracy while also ensuring efficient utilization of computing resources, and can flexibly adapt to the diverse needs of different industries and scenarios.

[0022] Third, it effectively solves the problem of result conflict in multimodal data fusion. This invention calculates the conflict degree between the analysis results of each modality. When the conflict degree exceeds a preset threshold, it automatically backs down and readjusts the fusion weights of each modality. It iterates until the conflict is eliminated or reduced to an acceptable range. This avoids the dilemma in traditional methods where different modal analysis results contradict each other but cannot be effectively resolved, and ensures the stability and reliability of the fusion results.

[0023] Fourth, it significantly reduces development and deployment costs. This invention is implemented in Java, fully leveraging the maturity and cross-platform capabilities of the Java ecosystem. It provides a standardized fusion analysis process and adaptive mechanism, eliminating the need for developers to repeatedly write underlying logic and debugging code for different data types and scenarios. This allows for rapid integration and deployment of multimodal data fusion analysis functions, significantly shortening the development cycle and reducing R&D costs. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the multimodal data fusion and analysis method provided by the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal data fusion and analysis device provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] To address the problems in existing technologies, this invention proposes a multimodal data fusion analysis method to achieve both high accuracy and computational efficiency. The multimodal data fusion analysis method is described below, as follows: Figure 1 As shown, including but not limited to the following steps: Step 110: Collect multimodal raw data, and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data.

[0028] In step 110, multimodal raw data refers to raw data originating from the same analysis object but with different modal types. Modal types include at least multiple types of text, images, audio, and video. During data collection, raw data can be obtained from various conventional data sources, such as reading from relational or non-relational databases, loading from locally stored files (e.g., log files, image files, audio files), or obtaining data from a remote server via a network interface. The collection method can be flexibly selected according to the actual application scenario. For example, in an intelligent customer service scenario, text messages, voice streams, and video streams from users can be received via real-time streaming; in a medical diagnosis scenario, medical record texts, medical image files, and consultation recordings can be retrieved in batches from the hospital information system.

[0029] After data acquisition, the raw data undergoes cleaning and standardization preprocessing. Cleaning eliminates noise and inconsistencies in the raw data, specifically including format unification, missing value imputation, and outlier removal. Format unification involves converting data from different sources and formats into a unified encoding format or data structure; for example, unifying images to JPEG format and fixed size, and audio to WAV format and fixed sampling rate. Missing value imputation involves filling in missing fields in the data, such as using mean imputation, mode imputation, or prediction imputation based on historical data. Outlier removal involves identifying, removing, or correcting data points that significantly deviate from the normal range, such as sensor readings exceeding reasonable numerical ranges or corrupted image files. Standardization, after cleaning, involves scaling or amplitude normalization of the data; for example, normalizing image pixel values ​​to the [0,1] range and normalizing audio signal amplitude to eliminate dimensional differences caused by different acquisition devices or conditions. After these processes, preprocessed multimodal target data is obtained, which has a unified format specification and numerical range and can be directly used for subsequent feature extraction.

[0030] Step 120: For each modality in the multimodal target data, use a feature extraction algorithm corresponding to the modality type to extract the feature vector of that modality.

[0031] In step 120, the preprocessed multimodal target data contains data of various modalities. The data of different modalities have essential differences in physical properties and information expression methods. Therefore, it is necessary to use an appropriate feature extraction algorithm for each modal type to transform the original data into feature vectors that can be numerically calculated by the computer.

[0032] For text modal data, a feature extraction method based on word embedding is adopted. Specifically, the text is first preprocessed by word segmentation and stop word removal, and then each word is mapped into a dense low-dimensional vector using a word embedding model. Finally, the entire text is converted into a fixed-dimensional text feature vector through weighted averaging or sequence aggregation.

[0033] For image modal data, a deep learning network is used for feature extraction. Specifically, the image data is input into a pre-trained convolutional neural network, which extracts low-level features such as edges, textures, and shapes, as well as high-level features such as object parts and scene semantics, layer by layer through multiple convolutional and pooling layers. Finally, a fixed-dimensional image feature vector is output from a specific layer of the network (such as a fully connected layer or a global pooling layer).

[0034] For audio modal data, acoustic feature parameters are extracted. Specifically, after framing and windowing the audio signal, a Fourier transform is performed to obtain the spectral information. Then, the spectrum is mapped to the Mel frequency scale using a Mel filter bank. Finally, the Mel frequency cepstral coefficients are obtained as audio feature vectors through discrete cosine transform.

[0035] For video modal data, visual and audio features can be extracted simultaneously. After decoding the video into image frame sequences and audio streams, image feature extraction methods and audio feature extraction methods are used to obtain the corresponding feature vectors. Then, the temporal frame-level features are aggregated into the overall video feature vector.

[0036] After the feature vectors of each modality are extracted, optional normalization processing can be performed on each feature vector to eliminate the differences in the dimensions and numerical ranges between different modal feature vectors, laying the foundation for subsequent fusion analysis.

[0037] Step 130: Perform adaptive fusion analysis. Based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch, perform fusion analysis on the feature vector to generate fusion features.

[0038] In step 130, adaptive fusion analysis means that the fusion process does not use a fixed fusion strategy and fixed modal weights, but dynamically adjusts the fusion method according to the actual quality and scale of the current data in order to achieve the optimal fusion effect.

[0039] Dynamic quality parameters for each modality are used to evaluate the reliability and information content of each modality in the current batch. In practice, these dynamic quality parameters include at least the noise level, missing data rate, and historical accuracy for each modality. The noise level measures the degree of noise interference affecting the modality data and can be determined by calculating the signal-to-noise ratio or statistically analyzing the proportion of abnormal signals. The missing data rate measures the proportion of missing information in the modality data and can be determined by statistically analyzing the proportion of missing fields or missing data blocks to the total data. Historical accuracy measures the performance of the modality in historical analysis tasks and can be determined by statistically analyzing the accuracy of single-modality analyses in past batches. These quality parameters can be calculated in real time or based on sliding window statistics.

[0040] The current batch of data to be processed refers to the number of data samples that need to be fused for analysis. In practice, the most suitable fusion method can be determined based on the data size. Specifically, when the data size is small, a computationally less demanding fusion method can be used to quickly obtain results; when the data size is large, a computationally more demanding but more accurate fusion method can be used; and when the data size is in the middle range, a compromise fusion method that balances efficiency and effectiveness can be adopted.

[0041] During the fusion analysis, the confidence level of each modality is first calculated based on the dynamic quality parameters of each modality's data. The confidence level reflects the reliability of that modality within the current batch of data. Then, the fusion weight of each modality is dynamically determined based on its confidence level. Modalities with higher confidence levels are assigned higher fusion weights to maximize their contribution to the fusion result; modalities with lower confidence levels are assigned lower fusion weights to mitigate their adverse effects. Simultaneously, initial weights preset according to industry type are used as a reference benchmark for adjusting the fusion weights, ensuring that the fusion process has a reasonable initial state for different industry scenarios.

[0042] After determining the fusion weights, a target fusion strategy is determined based on the data scale of the current batch. This strategy defines the specific combination method of each modality's feature vectors during the fusion process. Then, based on the determined target fusion strategy and dynamically allocated fusion weights, cross-modal semantic alignment and fusion processing are performed on the feature vectors of each modality. Cross-modal semantic alignment refers to mapping feature vectors from different modalities to a unified semantic space, enabling information from different modalities to interact and combine within the same semantic framework. The fusion processing, based on the alignment, combines the feature vectors of each modality according to the determined target fusion strategy to generate a comprehensive fusion feature that simultaneously contains the core information of each modality's data.

[0043] After obtaining the fused features, optional dimensional reduction or feature selection processing can be performed on the fused features to remove redundant information, reduce feature dimensions, and improve the efficiency of subsequent inference and analysis.

[0044] Step 140: Input the fusion features into a preset intelligent analysis model for reasoning to obtain multimodal fusion analysis results.

[0045] In step 140, the preset intelligent analysis model is a pre-trained machine learning model or deep learning model used to perform a specific analysis task. The type of model is related to the specific analysis task; for example, a classification model can be used in a classification task, a regression model in a regression task, and a sequence labeling model in a sequence labeling task. The model training process is completed offline, using labeled historical multimodal data samples. Supervised learning is used to enable the model to learn the mapping relationship between fused features and the final analysis results.

[0046] During the inference phase, the fusion features generated in step 130 are input into the preset intelligent analysis model. The model performs forward computation on the fusion features and outputs the probability distribution or predicted value corresponding to each candidate analysis result. Then, the final multimodal fusion analysis result is determined based on the model's output. For example, in a classification task, the category with the highest probability is taken as the final analysis result; in a regression task, the predicted value output by the model is taken as the final analysis result.

[0047] The final multimodal fusion analysis results can be further output to downstream systems or displayed to users. The output method can be flexibly selected according to actual application needs, such as outputting in the form of structured text reports for easy manual reading and archiving; outputting in the form of visual charts for easy intuitive display of analysis conclusions; or outputting in the form of API interfaces for easy docking and integration with third-party systems.

[0048] The technical solution of the present invention has been described in detail above with reference to specific embodiments. However, the scope of protection of the present invention is not limited to the specific embodiments described above. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0049] As a further optional embodiment, the step of performing adaptive fusion analysis, which involves fusing the feature vectors based on the dynamic quality parameters of each modality and the size of the current batch of data to be processed, to generate fused features, specifically includes: The confidence level of each modality is calculated based on the noise level, missing rate, and historical accuracy of each modality data. The fusion weight of each modality is dynamically determined based on the confidence level and the preset initial weight of industry type. Determine the target fusion strategy based on the size of the current batch of data to be processed; Based on the target fusion strategy and the fusion weights, cross-modal semantic alignment and fusion processing are performed on the feature vectors of each modality to obtain fused features.

[0050] In this embodiment, the dynamic quality parameters of each modality data include at least three dimensions: noise level, missing rate, and historical accuracy. The noise level measures the degree of noise interference affecting the current modality data; the missing rate measures the proportion of missing information in the data; and the historical accuracy reflects the reliability of the modality in historical analysis tasks. Through comprehensive calculation of these three dimensions, a confidence value for each modality in the current batch of data is obtained, which quantitatively reflects the reliability of the modality data. After obtaining the confidence value for each modality, it is combined with a preset initial weight for the industry type to dynamically determine the fusion weight of each modality. The initial weight for the industry type is a pre-set benchmark weight based on the industry domain of the current application scenario. For example, in a medical diagnosis scenario, the initial weight for image modalities (such as medical images) can be set to a higher value, while in an intelligent customer service scenario, the initial weight for text modalities can be set to a higher value. The core of dynamically determining fusion weights lies in: using preset initial weights for industry types as a base, dynamically adjusting the initial weights based on confidence levels—increasing the fusion weight of the corresponding modality when the confidence level is higher than expected, and decreasing the fusion weight of the corresponding modality when the confidence level is lower than expected. This ensures that the final determined fusion weights maintain the rationality of industry experience while flexibly adapting to the actual quality of the current batch of data. The fusion weights of each modality can be further normalized so that the sum of the fusion weights of all modalities is 1.

[0051] Then, the target fusion strategy is determined based on the size of the data to be processed in the current batch. Specifically, the size of the data to be processed in the current batch directly affects the computational complexity and accuracy requirements of the fusion analysis. The optimal fusion strategy varies depending on the data size. The system automatically selects an appropriate fusion strategy based on the number of samples in the data to be processed in the current batch. Specifically, the target fusion strategy can be determined from a variety of preset fusion strategies, such as, but not limited to, weighted fusion, feature concatenation fusion, and deep learning fusion. Among them, weighted fusion refers to weighted averaging or weighted summing of the feature vectors of each modality, which has the least computational cost but relatively limited fusion capability; feature concatenation fusion refers to directly concatenating the feature vectors of each modality to form a high-dimensional feature vector, which has moderate fusion capability; deep learning fusion refers to using a deep neural network structure to perform nonlinear interactive modeling of the features of each modality, which has the strongest fusion capability but also the largest computational cost. By automatically adapting the fusion strategy according to the data size, an optimal balance can be achieved between ensuring analytical accuracy and computational efficiency.

[0052] Finally, based on the determined target fusion strategy and dynamically determined fusion weights, cross-modal semantic alignment and fusion processing are performed on the feature vectors of each modality to obtain fused features. Specifically, cross-modal semantic alignment refers to mapping feature vectors of different modalities from their respective independent feature spaces to a unified semantic space. Since the feature vectors of different modalities have essential differences in physical meaning, numerical range, and distribution characteristics, direct fusion will produce semantic mismatch problems, so semantic alignment processing is required first. After completing semantic alignment, the feature vectors of each modality are fused according to the target fusion strategy: if the target fusion strategy is weighted fusion, the feature vectors of each modality are weighted and summed or weighted averaged according to the fusion weights of each modality to obtain fused features; if the target fusion strategy is feature concatenation fusion, the feature vectors of each modality are concatenated end to end to form high-dimensional fused features, and the feature vectors can be scaled according to the fusion weights of each modality during the concatenation process; if the target fusion strategy is deep learning fusion, the feature vectors of each modality and their weights are nonlinearly fused through the multi-layer structure of a deep neural network. During the fusion process, an attention mechanism can be further introduced to dynamically adjust the contribution intensity of each modality feature in the fusion. After the above processing, a comprehensive fusion feature is generated. This feature not only contains the core semantic information of each modality data, but also fully considers the actual quality of each modality data and the characteristics of the current batch data through weight allocation and strategy selection, providing high-quality input for subsequent inference analysis.

[0053] The adaptive fusion analysis method described above effectively overcomes the technical shortcomings of traditional fixed-weight, fixed-strategy fusion methods, which suffer from decreased analytical performance when data quality fluctuates or data scale changes.

[0054] As a further optional embodiment, after the step of performing cross-modal semantic alignment and fusion processing on the feature vectors of each modality based on the target fusion strategy and the fusion weights to obtain fused features, the method further includes: Based on each modal data in the multimodal target data, the modal analysis results are obtained by using the corresponding single-modal analysis model. Calculate the degree of conflict between the modal analysis results; If the conflict level exceeds a preset threshold, the process will revert to dynamically determining the fusion weights of each modality and readjusting the fusion weights of each modality. Based on the readjusted fusion weights, cross-modal semantic alignment and fusion processing is performed again until the conflict degree is reduced to below the preset threshold.

[0055] In this embodiment, before or after obtaining the fusion features, independent analysis results for each modality are obtained based on the modal data in the multimodal target data using their respective single-modal analysis models. Each modality is configured with a corresponding single-modal analysis model, which is specifically trained for the data characteristics of that modality and can independently output the analysis conclusions under that modality. For example, the text modality corresponds to a text classification model, the image modality corresponds to an image recognition model, and the audio modality corresponds to an audio classification model, etc. Each single-modal analysis model runs independently and outputs the probability distribution of each candidate analysis result under that modality.

[0056] After obtaining independent analysis results for each modality, the degree of conflict between the analysis results is calculated. The degree of conflict is used to quantitatively measure the degree of discrepancy between the analysis results of different modalities. When the analysis results of different modalities tend to be consistent, the degree of conflict is low, indicating that the information from each modality corroborates each other, and the reliability of the fusion result is high. When there are significant discrepancies between the analysis results of different modalities, the degree of conflict is high, indicating that there are contradictions between the information from each modality, and the fusion result may have biases. Specifically, the degree of conflict can be calculated by comparing the differences between the probability distributions of the analysis results of each modality, for example, by calculating the sum of the differences in the probability distributions between all pairs of modalities. If the probability distributions of the outputs of each modality are highly consistent, the degree of conflict approaches zero; if the probability distributions of the outputs of each modality differ significantly, the degree of conflict increases accordingly.

[0057] A preset threshold is set as the upper limit for the tolerance of conflict level. When the calculated conflict level does not exceed the preset threshold, it indicates that the discrepancies between the analysis results of each modality are within an acceptable range, and the current fused features can be directly used for subsequent inference analysis. When the conflict level exceeds the preset threshold, it indicates that there are serious semantic conflicts between the modalities, and at this time, the conflict resolution mechanism needs to be activated.

[0058] The conflict resolution mechanism specifically involves reverting to the process of dynamically determining the fusion weights of each modality and readjusting these weights. Specifically, excessive conflict indicates that the current fusion weight allocation for each modality is unreasonable, resulting in a failure to effectively reconcile discrepancies between modalities during the fusion process. Therefore, it is necessary to reassess the reliability and contribution of each modality based on the conflict situation of the analysis results and correct the fusion weights accordingly. For example, for modalities that are generally inconsistent with the analysis results of other modalities, their fusion weights are reduced; for modalities that are highly consistent with the analysis results of other modalities, their fusion weights are maintained or increased. The corrected fusion weights can more accurately reflect the actual reliability of each modality in the current batch of data.

[0059] After readjusting the fusion weights, cross-modal semantic alignment and fusion processing are performed again based on the readjusted fusion weights to obtain fused features again. The new fused features, due to the use of the corrected fusion weights, can more reasonably integrate information from various modalities, and the semantic conflicts between modalities are effectively mitigated.

[0060] The process of backtracking, readjusting fusion weights, and re-performing cross-modal semantic alignment and fusion can be performed iteratively. In each iteration, the conflict degree between the analysis results of each modality is recalculated until the conflict degree is reduced to below a preset threshold. Through this iterative conflict resolution mechanism, the system can automatically correct when there are discrepancies in the information of each modality, ensuring that the final output fused features have high semantic consistency and reliability.

[0061] Through the above-mentioned conflict detection and iterative correction process, the common cross-modal result conflict problem in multimodal data fusion is effectively solved, avoiding the dilemma of contradictory analysis results of different modalities in traditional methods that cannot be effectively resolved, and significantly improving the stability and reliability of the fusion results.

[0062] As a further optional embodiment, the confidence level of each modality is calculated using the following formula:

[0063] in, For each modality, Let m be the noise level of the m-th mode. Let m be the missing rate of the m-th mode. Let be the historical accuracy of the m-th mode.

[0064] In this embodiment, the confidence calculation formula comprehensively evaluates the reliability of each modality of data from three dimensions: noise level, noise level, and other factors. N m The value ranges from 0 to 1, reflecting the degree of noise interference in the modality data; a larger value indicates more severe noise. Missing rate. M m The value range is also from 0 to 1, reflecting the proportion of missing information in the modality data; the larger the value, the more severe the data loss. (1) N m This reflects the purity of the data, that is, the proportion that is not contaminated by noise; (1) M m This reflects the completeness of the data, i.e., the proportion of data without missing information. Historical accuracy A m The value ranges from 0 to 1, reflecting the performance of the mode in the historical analysis task. The larger the value, the higher the accuracy and reliability of the historical analysis of the mode.

[0065] The confidence level is obtained by multiplying the above three factors. C m The physical meaning of confidence level is: the comprehensive degree of reliability of a modality's data in the current batch analysis, considering the three factors of data purity, data completeness, and historical reliability. A higher confidence level indicates better data quality and greater analytical value for that modality; a lower confidence level indicates poorer data quality and lower reference value in fusion analysis.

[0066] In the specific implementation process, noise level N m It can be determined in various ways. For example, for image modalities, it can be determined by calculating the signal-to-noise ratio or assessing the blurriness of the image; for audio modalities, it can be determined by detecting the power proportion of ambient noise; for text modalities, it can be determined by detecting spelling error rate or grammatical anomaly rate. Missing rate M m This can be determined by statistically analyzing the proportion of missing fields or missing data blocks in the modality data out of all data that should be collected. For example, this could be the proportion of missing pixel blocks in an image or the proportion of missing fields in text. Historical accuracy. A m The accuracy rate of this modality is determined by statistically analyzing the single-modality analysis results in historical fusion analysis tasks. A sliding window approach can be used to dynamically update the accuracy rate, only counting the historical accuracy rate in the most recent batches or the most recent period, to reflect the performance of this modality in recent tasks.

[0067] After calculating the confidence scores for each modality, the system combines these scores with preset initial weights for industry types to dynamically determine the fusion weights for each modality. The principle for determining the fusion weights is as follows: using the initial weights for industry types as a baseline and the confidence scores as a correction coefficient—modalities with higher confidence scores have their fusion weights increased, while modes with lower confidence scores have their fusion weights decreased. The final determined fusion weights for each modality are then normalized so that the sum of the fusion weights for all modalities equals 1.

[0068] By using the aforementioned confidence calculation formula and method, the system can evaluate the reliability of each modality in the current batch in real time and quantitatively, providing accurate data support for subsequent dynamic weight allocation and adaptive fusion, ensuring that the fusion analysis process can be flexibly adjusted according to the actual data quality, and significantly improving the accuracy and robustness of the fusion analysis.

[0069] As a further optional embodiment, the step of performing cross-modal semantic alignment and fusion processing on the feature vectors of each modality based on the target fusion strategy and the fusion weights to obtain fused features specifically includes: The feature vectors of different modalities are mapped to a unified semantic space to achieve cross-modal semantic alignment; Based on the cross-modal attention mechanism, the correlation degree between different modal features is calculated, and the attention weight of each modal feature in the fusion process is determined based on the correlation degree. Based on the fusion weight and the attention weight, the feature vectors of each modality are weighted and fused to generate the fused feature.

[0070] In this embodiment, feature vectors from different modalities are mapped to a unified semantic space to achieve cross-modal semantic alignment. Specifically, since feature vectors from different modalities originate from different types of data, they differ fundamentally in feature dimension, numerical range, and semantic expression level. For example, text feature vectors are typically located in word embedding semantic space, reflecting the semantic information of words or sentences; image feature vectors are typically located in the feature space of convolutional neural networks, reflecting the visual semantic information of images; and audio feature vectors are located in acoustic feature space, reflecting the physical and perceptual characteristics of sound. Feature vectors in these different spaces do not possess direct comparability and composability. Direct fusion would lead to a deterioration in the fusion effect due to semantic mismatch. Therefore, this embodiment uses cross-modal semantic alignment processing to map feature vectors from different modalities from their respective original feature spaces to a common, unified semantic space. In this unified semantic space, semantically similar content in different modalities has similar vector representations, thereby making the feature vectors of different modalities comparable and composable, laying a semantic foundation for subsequent fusion processing.

[0071] Secondly, based on the cross-modal attention mechanism, the correlation degree between different modal features is calculated, and the attention weight of each modal feature in the fusion process is determined according to the correlation degree. Specifically, the cross-modal attention mechanism is a technique for dynamically modeling the correlation between different modal features. In the cross-modal attention mechanism, features of one modality are used as query objects, and features of other modalities are used as key objects. By calculating the similarity or compatibility score between the query and the key, the correlation degree between that modality and other modalities is obtained. The correlation degree reflects the degree of semantic interrelation between different modalities—the higher the correlation degree, the stronger the correlation between the two modalities in the current semantic context, and the greater the complementarity of their information; the lower the correlation degree, the weaker the correlation between the two modalities in the current semantic context, and the more redundant or irrelevant information exists between them. After calculating the correlation degree between each modality, the attention weight of each modal feature in the fusion process is further determined based on the correlation degree. The principle for allocating attention weights is as follows: modalities with high relevance to other modalities receive higher attention weights in the fusion process, indicating that the modality has higher information value in the current context; modalities with low relevance to other modalities receive lower attention weights in the fusion process. The attention weights determined through the cross-modal attention mechanism enable dynamic capture of semantic relationships between modalities, allowing the fusion process to adaptively adjust the contribution of each modality based on the specific semantic content of the current data.

[0072] Finally, based on the fusion weight and the attention weight, the feature vectors of each modality are weighted and fused to generate the fused feature. Specifically, the fusion weight is determined after calculating the confidence level based on the dynamic quality parameters (noise level, missing rate, historical accuracy, etc.) of each modality's data, reflecting the reliability of the data quality of each modality itself; the attention weight is determined after calculating the correlation between the features of each modality based on the cross-modal attention mechanism, reflecting the information relevance of each modality in the current semantic context. The two weights evaluate the importance of each modality from different dimensions and complement each other. This embodiment combines the above two weights to jointly act on the fusion process of the feature vectors of each modality. In specific fusion, for the feature vector of each modality, a comprehensive weight is determined based on its fusion weight and attention weight, and then the feature vectors of each modality are weighted and combined according to the comprehensive weight to generate the fused feature. The specific method of weighted combination can be determined according to the target fusion strategy: if the target fusion strategy is weighted fusion, then the feature vectors of each modality are summed or averaged with weights; if the target fusion strategy is feature concatenation fusion, then the feature vectors of each modality are scaled according to the comprehensive weights before concatenation; if the target fusion strategy is deep learning fusion, then the comprehensive weights are used as the initial weights or gating signals of the corresponding modal input channels in the deep neural network, participating in the nonlinear fusion calculation of the network. Through the above dual-weight fusion processing, the generated fusion features fully consider the quality and reliability of each modality data, and also take into account the information relevance of each modality in the current semantic context, providing high-quality input for subsequent intelligent analysis and reasoning.

[0073] As a further optional embodiment, the step of collecting multimodal raw data and cleaning and standardizing the multimodal raw data to obtain preprocessed multimodal target data specifically includes: Collect multimodal raw data from at least one data source, including databases, local files, or network interfaces; The multimodal raw data is cleaned, including at least one of data format standardization, missing value filling, and outlier removal; The cleaned data is standardized to generate the preprocessed multimodal target data.

[0074] In this embodiment, multimodal raw data is collected from at least one data source, including databases, local files, and network interfaces. Specifically, database sources include relational databases (such as MySQL and Oracle) and non-relational databases (such as MongoDB and HBase), suitable for scenarios storing structured or semi-structured data, such as reading user profile data from a customer service system's database. Local file sources include various formats of locally stored files, such as log files, image files (such as JPEG and PNG), audio files (such as WAV and MP3), video files (such as MP4), and document files (such as TXT and PDF), suitable for batch import or offline analysis scenarios, such as loading historical medical image data from a local disk. Network interface sources include RESTful APIs called via HTTP / HTTPS protocols, real-time data streams received via WebSocket, or data channels subscribed to via message queues (such as Kafka and RabbitMQ), suitable for real-time or near-real-time analysis scenarios, such as obtaining user message streams from an intelligent customer service system in real time via API. These various data sources can be used individually or in combination depending on the actual application scenario. During the data acquisition process, the system can automatically adapt to the corresponding data reading method based on the data source's transmission protocol and data format, enabling unified access to multi-source heterogeneous data. For example, in intelligent customer service scenarios, it can receive users' text messages, voice streams, and video streams in real time through network interfaces, while simultaneously reading users' identity information and historical interaction records from the database; in medical diagnosis scenarios, it can retrieve structured medical record data from the database, load medical image files from the local file system, and obtain remote consultation recordings through network interfaces.

[0075] The collected multimodal raw data undergoes cleaning processing, including at least one of the following: data format unification, missing value imputation, and outlier removal. Format unification refers to converting data from different sources and in different formats into a unified encoding format or data structure. For example, unifying images of different formats into JPEG format and scaling them to a fixed size, unifying audio with different sampling rates into WAV format with a 16kHz sampling rate, and unifying text with different character encodings into UTF-8 encoding. Missing value imputation refers to completing missing fields or information in the data. For example, for missing text fields, empty strings or context-based prediction imputation are used; for missing image regions, neighbor pixel interpolation or generative model-based imputation are used; for missing audio segments, silence imputation or smooth interpolation based on preceding and following frames is used. Outlier removal refers to identifying, removing, or correcting data points that significantly deviate from the normal range. For example, detecting and removing corrupted image files, truncating audio signals exceeding reasonable amplitude ranges, and removing text records that clearly do not conform to format specifications. The above cleaning operations can be flexibly selected based on the data type and quality. For example, for high-quality data sources, only format standardization may be performed without filling in missing values; for low-quality data sources, a combination of cleaning methods is required. In practice, the system can pre-configure cleaning rules and quality standards for each modality of data, and the cleaning process will be automatically executed according to the configured rules.

[0076] The cleaned data is standardized to generate the preprocessed multimodal target data. Specifically, standardization involves further scaling or amplitude normalization of the data after cleaning to eliminate dimensional and numerical range differences caused by different acquisition devices, acquisition conditions, or data sources. For image data, standardization includes normalizing pixel values ​​from the [0,255] interval to the [0,1] interval, and Z-score standardization can be performed based on the mean and standard deviation of the training data. For audio data, standardization includes amplitude normalization, power normalization, or root mean square normalization to ensure consistent loudness levels across different audio samples. For text data, standardization includes converting the text to lowercase, removing redundant whitespace characters, and performing stemming or word form restoration. For numerical data, standardization includes scaling the values ​​of each feature dimension to a standard normal distribution with a mean of 0 and a variance of 1, or scaling to the [0,1] interval. Through the above standardization process, different modal data have a unified numerical range and distribution characteristics, which enables subsequent feature extraction and fusion analysis to be carried out on a consistent data benchmark, avoiding numerical calculation instability or fusion deviation caused by differences in units.

[0077] After the above data collection, cleaning and standardization processes, the preprocessed multimodal target data is generated. This target data has a unified format specification and numerical range, and can be directly used for subsequent feature extraction operations for each modality type.

[0078] As a further optional embodiment, the step of inputting the fused features into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results specifically includes: The fused features are input into a preset intelligent analysis model, which is a traditional machine learning model or a deep learning model. The intelligent analysis model is used to infer the fused features and output the probability distribution corresponding to each candidate analysis result. The final multimodal fusion analysis result is determined based on the probability distribution.

[0079] In this embodiment, the fused features are input into a preset intelligent analysis model, which can be a traditional machine learning model or a deep learning model. Specifically, the preset intelligent analysis model is a pre-trained computational model used to perform a specific analysis task. The type of model is related to the actual analysis task objective; for example, a classification model can be used in a classification task, a regression model in a regression task, a sequence labeling model in a sequence labeling task, and a ranking model or a matching model in a retrieval task. The model training process is completed offline. The training data consists of labeled historical multimodal fused feature samples and their corresponding real analysis result labels. Supervised learning is used to enable the model to learn the mapping relationship between the fused features and the final analysis result. The model type can be flexibly configured according to the specific needs of the application scenario: when the number of training samples is small or the interpretability of the model is high, traditional machine learning models, such as support vector machines, random forests, and logistic regression, can be used; when the number of training samples is sufficient and the analysis accuracy is high, deep learning models, such as fully connected neural networks, convolutional neural networks, and Transformer networks, can be used. All the models mentioned above are pre-trained and deployed in the system, and are directly loaded and invoked during the inference phase. The system can call different types of intelligent analysis models through a unified model interface without needing to know the specific implementation details of the underlying models, thus achieving seamless invocation of traditional machine learning models and deep learning models.

[0080] The intelligent analysis model infers from the fused features and outputs the probability distributions corresponding to each candidate analysis result. Specifically, the fused features generated in step 130 are used as input data and passed to a preset intelligent analysis model. If the intelligent analysis model is a traditional machine learning model (such as a support vector machine or random forest), the model performs discrimination calculations on the fused features, outputs the decision scores or probability estimates corresponding to each candidate category, and obtains the probability distributions of each candidate analysis result after normalization. If the intelligent analysis model is a deep learning model (such as a fully connected neural network or Transformer network), the fused features are sequentially passed through each hidden layer of the model for forward propagation calculation, and finally the output layer converts the network output into the probability distributions of each candidate analysis result through the Softmax function or the Sigmoid function. Each component in the probability distribution corresponds to a candidate analysis result, and the component value represents the probability of the candidate analysis result being true. The sum of all components is 1. For example, in the intent recognition task in an intelligent customer service scenario, the candidate analysis results may include multiple categories such as "complaint," "inquiry," "suggestion," and "return / exchange." The model outputs a probability distribution vector with the same number of candidate categories, and each component represents the probability that the current user's request belongs to each category.

[0081] The final multimodal fusion analysis result is determined based on the probability distribution. Specifically, after obtaining the probability distribution corresponding to each candidate analysis result, the final output result is determined from the candidate analysis results according to a preset decision rule. The most common decision rule is the maximum probability rule, which determines the candidate category with the highest probability value in the probability distribution as the final multimodal fusion analysis result. For example, in intelligent customer service intent recognition, if the probability of the "complaint" category is 0.78, the probability of the "consultation" category is 0.15, and the probability of the "suggestion" category is 0.07, then "complaint" is determined as the final user request type. In addition to the maximum probability rule, other decision rules can be adopted according to the needs of specific application scenarios. For example, when the highest probability is lower than a certain confidence threshold, the conclusion of "uncertain" or "requires manual review" can be output, or the top-ranked candidate results can be output for further processing by the downstream system. In regression tasks, the model outputs a continuous value rather than a probability distribution. In this case, the value can be directly used as the final multimodal fusion analysis result. In this way, step 140 transforms the fused features into specific analytical conclusions, thus realizing a complete closed loop from multimodal data input to final analytical result output.

[0082] The final multimodal fusion analysis results can be further output to downstream business systems or displayed to end users. The output method can be flexibly selected according to actual application needs. For example, it can be output as a structured text report for manual reading and archiving, as a visual chart to intuitively display the analysis conclusions, or as JSON format via an API interface for integration with third-party systems. The system can automatically select one or more output methods based on preset output configurations, enabling flexible distribution and application of analysis results.

[0083] The multimodal data fusion and analysis device provided by the present invention will be described below, such as... Figure 2 As shown, the multimodal data fusion analysis device described below and the multimodal data fusion analysis method described above can be referred to in correspondence.

[0084] A multimodal data fusion and analysis device, comprising: The data acquisition module 210 is used to acquire multimodal raw data and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data. Feature extraction module 220 is used to extract the feature vector of each modality in the multimodal target data using a feature extraction algorithm corresponding to the modality type; The feature fusion module 230 is used to perform adaptive fusion analysis, and to perform fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch, so as to generate fused features; The feature reasoning module 240 is used to input the fused features into a preset intelligent analysis model for reasoning to obtain multimodal fusion analysis results.

[0085] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a multimodal data fusion analysis method, which includes: Collect multimodal raw data, and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data; For each modality in the multimodal target data, a feature vector for that modality is extracted using a feature extraction algorithm corresponding to the modality type; Perform adaptive fusion analysis, and perform fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch to generate fusion features; The fused features are input into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results.

[0086] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the multimodal data fusion analysis method provided by the above methods, the method comprising: Collect multimodal raw data, and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data; For each modality in the multimodal target data, a feature vector for that modality is extracted using a feature extraction algorithm corresponding to the modality type; Perform adaptive fusion analysis, and perform fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch to generate fusion features; The fused features are input into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results.

[0088] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the multimodal data fusion analysis method provided by the methods described above, the method comprising: Collect multimodal raw data, and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data; For each modality in the multimodal target data, a feature vector for that modality is extracted using a feature extraction algorithm corresponding to the modality type; Perform adaptive fusion analysis, and perform fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch to generate fusion features; The fused features are input into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data fusion and analysis method, characterized in that, include: Collect multimodal raw data, and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data; For each modality in the multimodal target data, a feature vector for that modality is extracted using a feature extraction algorithm corresponding to the modality type; Perform adaptive fusion analysis, and perform fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch to generate fusion features; The fused features are input into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results.

2. The multimodal data fusion and analysis method according to claim 1, characterized in that, The step of performing adaptive fusion analysis, which involves fusing the feature vectors based on the dynamic quality parameters of each modality and the size of the current batch of data to be processed, to generate fused features, specifically includes: The confidence level of each modality is calculated based on the noise level, missing rate, and historical accuracy of each modality data. The fusion weight of each modality is dynamically determined based on the confidence level and the preset initial weight of industry type. Determine the target fusion strategy based on the size of the current batch of data to be processed; Based on the target fusion strategy and the fusion weights, cross-modal semantic alignment and fusion processing are performed on the feature vectors of each modality to obtain fused features.

3. The multimodal data fusion and analysis method according to claim 2, characterized in that, After the step of performing cross-modal semantic alignment and fusion processing on the feature vectors of each modality based on the target fusion strategy and the fusion weights to obtain fused features, the method further includes: Based on each modal data in the multimodal target data, the modal analysis results are obtained by using the corresponding single-modal analysis model. Calculate the degree of conflict between the modal analysis results; If the conflict level exceeds a preset threshold, the process will revert to dynamically determining the fusion weights of each modality and readjusting the fusion weights of each modality. Based on the readjusted fusion weights, cross-modal semantic alignment and fusion processing is performed again until the conflict degree is reduced to below the preset threshold.

4. The multimodal data fusion and analysis method according to claim 2, characterized in that, The confidence level of each mode is calculated using the following formula: in, For each modality, Let m be the noise level of the m-th mode. Let m be the missing rate of the m-th mode. Let be the historical accuracy of the m-th mode.

5. The multimodal data fusion analysis method according to claim 2, characterized in that, The step of performing cross-modal semantic alignment and fusion processing on the feature vectors of each modality based on the target fusion strategy and the fusion weights to obtain fused features specifically includes: The feature vectors of different modalities are mapped to a unified semantic space to achieve cross-modal semantic alignment; Based on the cross-modal attention mechanism, the correlation degree between different modal features is calculated, and the attention weight of each modal feature in the fusion process is determined based on the correlation degree. Based on the fusion weight and the attention weight, the feature vectors of each modality are weighted and fused to generate the fused feature.

6. The multimodal data fusion analysis method according to claim 1, characterized in that, The step of collecting multimodal raw data and cleaning and standardizing the multimodal raw data to obtain preprocessed multimodal target data specifically includes: Collect multimodal raw data from at least one data source, including databases, local files, or network interfaces; The multimodal raw data is cleaned, including at least one of data format standardization, missing value filling, and outlier removal; The cleaned data is standardized to generate the preprocessed multimodal target data.

7. The multimodal data fusion analysis method according to claim 1, characterized in that, The step of inputting the fused features into a preset intelligent analysis model for inference to obtain multimodal fusion analysis results specifically includes: The fused features are input into a preset intelligent analysis model, which is a traditional machine learning model or a deep learning model. The intelligent analysis model is used to infer the fused features and output the probability distribution corresponding to each candidate analysis result. The final multimodal fusion analysis result is determined based on the probability distribution.

8. A multimodal data fusion and analysis device, characterized in that, include: The data acquisition module is used to acquire multimodal raw data and perform cleaning and standardization preprocessing on the multimodal raw data to obtain preprocessed multimodal target data. The feature extraction module is used to extract the feature vector of each modality from the multimodal target data using a feature extraction algorithm corresponding to the modality type. The feature fusion module is used to perform adaptive fusion analysis, which performs fusion analysis on the feature vectors based on the dynamic quality parameters of each modality data and the size of the data to be processed in the current batch, and generates fused features. The feature reasoning module is used to input the fused features into a preset intelligent analysis model for reasoning to obtain multimodal fusion analysis results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal data fusion analysis method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal data fusion analysis method as described in any one of claims 1 to 7.

Images