Intelligent system integration method based on multi-modal data fusion

Abstract

The invention discloses an intelligent system integration method based on multi-modal data fusion. The method comprises the following steps: S1, acquiring a plurality of data sources of different modals; s2, preprocessing the original data in the plurality of data sources of different modalities to generate preprocessed data; s3, based on the preprocessed data, mining a potential association relationship between modal data, and generating an inter-modal association rule set; s4, dynamically adjusting a data fusion strategy according to the inter-modal association rule set, and generating an adjusted fusion strategy; s5, according to the adjusted fusion strategy, performing weighted fusion on the modal data to generate integrated data after fusion; s6, processing the integrated data after fusion, and generating integrated data; and S7, in the operation process of the intelligent system, updating the association rule in real time based on the integrated data and the new data source input. The method has the advantages of high flexibility, high efficiency, high adaptability and low data processing cost.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent systems, and in particular to an intelligent system integration method based on multimodal data fusion. Background Art

[0002] With the continuous advancement of information technology and the widespread application of artificial intelligence, intelligent systems have been widely used and promoted in many fields. Especially in terms of multimodal data fusion, how to effectively integrate data from different sources and different types to enhance the system's comprehensive decision-making ability has become an important research topic in the current research and development of intelligent systems. Multimodal data fusion can provide a more comprehensive perspective and decision support by integrating information from multiple data sources such as vision, speech, and sensors. However, existing intelligent system integration methods face a number of technical challenges.

[0003] In the existing technology, traditional multimodal data fusion methods mostly rely on static fusion rules and fixed data processing procedures. These methods usually use deep learning models or predefined rules to fuse data, but when faced with complex associations between different modal data, they are often unable to flexibly adapt and generate the optimal fusion strategy. Specifically, the existing technology has the following shortcomings:

[0004] 1. Lack of dynamic adaptability: Existing fusion methods usually use fixed models or rules and cannot automatically adjust the fusion strategy according to new data sources or data changes in real time, resulting in poor performance when processing new and changing data sources.

[0005] 2. Information redundancy and conflict problems: When processing multimodal data, traditional methods are often unable to effectively identify and process the redundancy and conflict between modal data, which can easily lead to inaccuracy or inconsistency in information fusion, thus affecting the final system decision.

[0006] 3. High dependence on labeled data: Most existing intelligent system integration methods rely on a large amount of labeled data for training, which poses a serious constraint on the data labeling cost and the availability of labeled data in practical applications. This is especially true in scenarios where data changes rapidly or diversifies. Traditional methods are unlikely to provide sufficient flexibility.

[0007] 4. Fixed nature of fusion strategy: The data fusion strategy in traditional methods is usually predefined by the deep learning network structure, lacks the ability to adaptively adjust to different data sources, and cannot dynamically optimize the fusion mode according to the correlation between different modal data.

[0008] Therefore, how to provide an intelligent system integration method based on multimodal data fusion is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0009] One purpose of the present invention is to propose an intelligent system integration method based on multimodal data fusion. The present invention makes full use of adaptive association rule mining technology and inter-modal association network framework, and describes in detail how to optimize the data fusion process by dynamically adjusting the fusion strategy, reduce information redundancy and conflict, and reduce dependence on a large amount of labeled data. It has the advantages of strong flexibility, high efficiency, strong adaptability and low data processing cost.

[0010] The intelligent system integration method based on multimodal data fusion according to an embodiment of the present invention comprises the following steps:

[0011] S1. Obtain multiple data sources of different modalities;

[0012] S2, preprocessing the raw data from multiple data sources of different modalities, including denoising and standardization, to generate preprocessed data;

[0013] S3. Based on the preprocessed data, an adaptive association rule mining algorithm is used to mine the potential association relationship between the preprocessed data of each modality and generate an inter-modality association rule set;

[0014] S4, dynamically adjusting the data fusion strategy according to the set of association rules between modalities, and generating an adjusted fusion strategy;

[0015] S5. According to the adjusted fusion strategy, weighted fusion is performed on the pre-processed data of each modality to generate fused comprehensive data;

[0016] S6, processing the fused comprehensive data to generate integrated data;

[0017] S7. During the operation of the intelligent system, based on the integrated data and new data source input, the set of inter-modality association rules is updated in real time to generate updated association rules, and the fusion strategy is readjusted according to the updated association rules.

[0018] Optionally, the S3 specifically includes:

[0019] S31, performing modal division on the preprocessed data to obtain independent modal data sets for each modal data, wherein the preprocessed data is represented as ,in Indicates independent modal data sets, n represents the total number of independent modal data in the modal data set, including All sample data in the independent modal data set, each sample data consists of a feature vector Composition, of which is the number of eigenvectors, and Indicates In the modal The value of the eigenvector;

[0020] S32, in each modality data set, select an appropriate subset of feature vectors, and calculate the correlation between the feature vectors in the selected feature vector subset, the correlation To reflect the characteristics and Features The relationship between: ;

[0021] in, and Respectively represent Features and The feature in The values ​​in the subset of eigenvectors, and Respectively Features and Features in the modal The mean value under is the number of eigenvector subsets;

[0022] S33, according to the correlation between the feature vectors, an adaptive association rule mining algorithm is used to set the support threshold and confidence threshold , mining the potential association rules between modal data and generating the association rule set between modalities , where each association rule Expressed as ,in and are two feature sets and satisfy the following conditions:

[0023] ;

[0024] in, for and The probability of simultaneous occurrence, For Derived The conditional probability of

[0025] S34, according to the generated set of association rules between modalities, by calculating the lifting degree of each rule To evaluate the strength of each association rule, we can get a set of association rules with higher strength: ;

[0026] in, and The feature sets and The marginal probability of , then it means and There is a strong positive correlation between them;

[0027] S35: Filter out association rules with stronger inter-modal dependencies from the association rule set with higher strength, and generate a final inter-modal association rule set. , where each association rule Indicates the relationship between modes.

[0028] Optionally, the S4 specifically includes:

[0029] S41, based on the inter-modality association rule set, calculate each association rule The corresponding weight value , where each weight value Reflects the contribution of the association rule to the modal data fusion process: ;

[0030] in, Association rules The degree of improvement, Association rules middle and The probability of simultaneous occurrence, and are other feature sets in the inter-modal association rule set, For the comprehensive contribution of all association rules, is the number of rules in the inter-modality association rule set;

[0031] S42. According to each association rule The weight value , calculate each modal data set Weights in data fusion : ;

[0032] in, For independent modality datasets The correlation value between the modalities in the association rules, is the weight value of the association rule, and n represents the total number of independent modal data in the modal data set;

[0033] S43. For each independent modality data set Weight Perform normalization to obtain the normalized weight value ;

[0034] S44, according to the normalized weight value , generate the adjusted fusion strategy; this strategy uses the weighted average method to calculate the normalized weight value of each modal data. Perform weighted fusion on modal data to obtain the adjusted fusion strategy : ;

[0035] Where n represents the total number of independent modal data in the modal data set;

[0036] S45. According to the adjusted fusion strategy, feature selection and dimensionality reduction processing of the modal data are performed, and the normalized weight value of each independent modal data set is adjusted in real time.

[0037] Optionally, the S5 specifically includes:

[0038] S51, based on the adjusted fusion strategy , for each modality dataset According to its normalized weight value Perform weighting to generate a weighted modal data set : ;

[0039] in, Indicates Independent modality datasets;

[0040] S52, for all weighted modal data Add and generate integrated data : ;

[0041] in, is the integrated data after fusion. is the weighted modal data set, n represents the total number of independent modal data in the modal data set;

[0042] S53. Comprehensive data after fusion Perform feature extraction and standardization to generate processed fusion data ;

[0043] S54, based on the processed fusion data , optimize the weight of each modal data in the fusion process, and adjust the fusion strategy in real time by evaluating the contribution of each modal data set;

[0044] S55: regenerate the weighted modality data set and update the comprehensive data according to the adjusted fusion strategy .

[0045] Optionally, the S53 specifically includes:

[0046] S531. Comprehensive data after fusion Perform feature extraction, select feature subsets, and measure the contribution of each feature to the feature subset by calculating the importance of the features contained in the feature subset: ;

[0047] in, For the Features The importance value of Features The variance of is the total number of features, Features The variance of

[0048] S532, based on the selected feature subset, the fused comprehensive data Standardization is performed so that the mean of each feature is , the standard deviation is , get the standardized data value ;

[0049] S533, data value after standardization The principal component analysis is performed on the dimensionality reduction operation. By calculating the characteristic covariance matrix and solving its eigenvalues ​​and eigenvectors, the principal components with the largest variance are selected and the data are mapped to these principal components to generate the data representation after dimensionality reduction. ;

[0050] S534, representing the data after dimension reduction , perform outlier detection on the reduced-dimensional data representation, remove outliers, and generate processed fused data .

[0051] Optionally, the S6 specifically includes:

[0052] S61, the integrated data after fusion De-noising is performed, and median filtering is used to remove high-frequency noise and retain valid signals. The denoised data is expressed as ;

[0053] S62. De-noised data Normalize the feature so that the value of each feature is within a uniform scale: ;

[0054] in, The denoised data Middle The sample in The dimension value of the feature, For the The mean of the features, For the The standard deviation of the feature, For the The weighting coefficient of the feature, For the The variance of the feature, is the normalized data value;

[0055] S63, performing feature selection on the normalized data values, and selecting a strongly correlated feature subset that is highly correlated with the final task;

[0056] S64, based on the selected strongly correlated feature subset Reconstruct the normalized data values ​​and generate integrated data by calculating the weighted average of each feature : ;

[0057] in, For the The selected features are in the strongly correlated feature subset The weight in For the The selected features are in the normalized data The value in is the number of features in the strongly correlated feature subset.

[0058] Optionally, the S7 specifically includes:

[0059] S71. During the operation of the intelligent system, new data source input is received , the new data source input includes new modality data and its related features;

[0060] S72. Input based on new data source , for existing integrated data Perform preliminary fusion processing to generate temporary fusion data : ;

[0061] in, For independent modality datasets In the new data source The weight in For the new data source data of each modality, For independent modality datasets in existing integrated data The weight of To integrate data Middle data of each modality, is the number of modes in the new data source, is the number of modes in the existing integrated data;

[0062] S73, based on the temporary fusion data , adaptively update the existing association rules, use incremental learning algorithms or other applicable methods to update the current association rule set Make real-time corrections to generate updated association rule sets : ;

[0063] in, Based on temporary fusion data and new data sources Association rules The function to update. is the updated association rule;

[0064] S74: According to the updated association rule set , recalculate the weight value of the modal data ;

[0065] S75: Based on the updated weight value , readjust the fusion strategy and generate a new fusion strategy .

[0066] The beneficial effects of the present invention are:

[0067] (1) The present invention adopts adaptive association rule mining technology and inter-modal association network framework to dynamically adjust data fusion strategy, thereby effectively solving the problems of information redundancy, information conflict and fixed fusion strategy in traditional multimodal data fusion methods. This method improves the flexibility and adaptability of the system when facing complex and variable data sources by automatically optimizing the fusion strategy according to the potential correlation between modal data.

[0068] (2) The present invention significantly reduces the cost and workload of data annotation by reducing the reliance on large amounts of labeled data and utilizing adaptive data fusion strategies and dynamic correlations between modalities. This enables the system to operate efficiently in an environment where data sources change rapidly, and is particularly suitable for processing large-scale, heterogeneous data sources.

[0069] (3) The present invention uses an adaptive association rule update mechanism to adjust the fusion strategy in real time according to the new data source input and optimize the weighted fusion process of modal data, thereby ensuring that the system can continuously adapt to changes in different data sources and improving the real-time, accuracy and stability of intelligent system integration. This dynamic adaptive mechanism significantly improves the system's ability to cope with different modal data inputs, especially in the fields of intelligent decision support and multimodal data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0071] Figure 1 This is a flow chart of the intelligent system integration method based on multimodal data fusion proposed by the present invention. DETAILED DESCRIPTION

[0072] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0073] refer to Figure 1 , an intelligent system integration method based on multimodal data fusion, comprising the following steps:

[0074] S1. Obtain multiple data sources of different modalities;

[0075] This implementation provides comprehensive data integration by acquiring multiple data sources of different modalities, allowing the system to obtain information from multiple dimensions. This process not only enriches the source of data, but also improves the diversity and comprehensiveness of data, thereby enhancing the intelligent system's ability to process diverse data in complex environments.

[0076] S2, preprocessing the raw data from multiple data sources of different modalities, removing noise and performing standardization to generate preprocessed data;

[0077] This implementation significantly improves the quality and consistency of data by preprocessing the raw data from multiple data sources of different modalities, removing noise and performing standardization. Denoising and standardization ensure the comparability of data from each modality in the same data space, thereby avoiding errors caused by noise interference and inconsistent data scales.

[0078] S3. Based on the preprocessed data, an adaptive association rule mining algorithm is used to mine the potential association relationship between the modal data and generate a set of inter-modal association rules;

[0079] In this implementation, S3 specifically includes:

[0080] S31, performing modal division on the preprocessed data to obtain independent modal data sets for each modal data, wherein the preprocessed data is represented as ,in Indicates independent modal data sets, n represents the total number of independent modal data in the modal data set, including All sample data in the independent modal data set, each sample data consists of a feature vector Composition, of which is the number of eigenvectors, and Indicates In the modal The value of the eigenvector;

[0081] S32, in each modality data set, select an appropriate subset of feature vectors, and calculate the correlation between the feature vectors in the selected feature vector subset, the correlation To reflect the characteristics and Features The relationship between: ;

[0082] in, and Respectively represent Features and The feature in The values ​​in the subset of eigenvectors, and Respectively Features and Features in the modal The mean value under is the number of eigenvector subsets;

[0083] S33, according to the correlation between the feature vectors, an adaptive association rule mining algorithm is used to set the support threshold and confidence threshold , mining the potential association rules between modal data and generating the association rule set between modalities , where each association rule Expressed as ,in and are two feature sets and satisfy the following conditions: ;

[0084] in, for and The probability of simultaneous occurrence, For Derived The conditional probability of

[0085] S34, according to the generated set of association rules between modalities, by calculating the lifting degree of each rule To evaluate the strength of each association rule, we can get a set of association rules with higher strength: ;

[0086] in, and The feature sets and The marginal probability of , then it means and There is a strong positive correlation between them;

[0087] S35: Filter out association rules with stronger inter-modal dependencies from the association rule set with higher strength, and generate a final inter-modal association rule set. , where each association rule Indicates the relationship between modes.

[0088] This embodiment uses an adaptive association rule mining algorithm based on preprocessed data to mine potential associations between modal data and generate an association rule set between modalities. This process can automatically discover deep associations between different modal data, avoiding the limitations and fixity of manually set rules.

[0089] S4, dynamically adjusting the data fusion strategy according to the set of association rules between modalities, and generating an adjusted fusion strategy;

[0090] In this implementation, S4 specifically includes:

[0091] S41, based on the inter-modality association rule set, calculate each association rule The corresponding weight value , where each weight value Reflects the contribution of the association rule to the modal data fusion process: ;

[0092] in, Association rules The degree of improvement, Association rules middle and The probability of simultaneous occurrence, and are other feature sets in the inter-modal association rule set, For the comprehensive contribution of all association rules, is the number of rules in the inter-modality association rule set;

[0093] S42. According to each association rule The weight value , calculate each modal data set Weights in data fusion : ;

[0094] in, For independent modality datasets The correlation value between the modalities in the association rules, is the weight value of the association rule, and n represents the total number of independent modal data in the modal data set;

[0095] S43. For each independent modality data set Weight Perform normalization to obtain the normalized weight value ;

[0096] S44, according to the normalized weight value , generate the adjusted fusion strategy; this strategy uses the weighted average method to calculate the normalized weight value of each modal data. Perform weighted fusion on modal data to obtain the adjusted fusion strategy : ;

[0097] Where n represents the total number of independent modal data in the modal data set;

[0098] S45. According to the adjusted fusion strategy, feature selection and dimensionality reduction processing of the modal data are performed, and the normalized weight value of each independent modal data set is adjusted in real time.

[0099] This implementation method dynamically adjusts the data fusion strategy according to the set of inter-modal association rules to generate an adjusted fusion strategy, thereby achieving flexible adaptation to the relationship between different modal data. This method can optimize the data fusion strategy in real time according to the association rules, avoids the limitations of fixed fusion modes, and improves the adaptability and intelligence level of the system when processing complex data sources.

[0100] S5. Perform weighted fusion on each modality data according to the adjusted fusion strategy to generate fused comprehensive data;

[0101] In this implementation, S5 specifically includes:

[0102] S51, based on the adjusted fusion strategy , for each modality dataset According to its normalized weight value Perform weighting to generate a weighted modal data set : ;

[0103] in, Indicates Independent modality datasets;

[0104] S52, for all weighted modal data Add and generate integrated data : ;

[0105] in, is the integrated data after fusion. is the weighted modal data set, n represents the total number of independent modal data in the modal data set;

[0106] S53. Comprehensive data after fusion Perform feature extraction and standardization to generate processed fusion data ;

[0107] S54, based on the processed fusion data , optimize the weight of each modal data in the fusion process, and adjust the fusion strategy in real time by evaluating the contribution of each modal data set;

[0108] S55: regenerate the weighted modality data set and update the comprehensive data according to the adjusted fusion strategy .

[0109] The S53 specifically includes:

[0110] S531. Comprehensive data after fusion Perform feature extraction, select feature subsets, and measure the contribution of each feature to the feature subset by calculating the importance of the features contained in the feature subset: ;

[0111] in, For the Features The importance value of Features The variance of is the total number of features, Features The variance of

[0112] S532, based on the selected feature subset, the fused comprehensive data Standardization is performed so that the mean of each feature is , the standard deviation is , get the standardized data value ;

[0113] S533, data value after standardization The principal component analysis is performed on the dimensionality reduction operation. By calculating the characteristic covariance matrix and solving its eigenvalues ​​and eigenvectors, the principal components with the largest variance are selected and the data are mapped to these principal components to generate the data representation after dimensionality reduction. ;

[0114] S534, representing the data after dimension reduction , perform outlier detection on the reduced-dimensional data representation, remove outliers, and generate processed fused data .

[0115] This implementation achieves effective integration of each modality data by weighted fusion of each modality data according to the adjusted fusion strategy to generate fused comprehensive data. Through weighted fusion, the system can dynamically adjust the contribution of each modality according to the relative importance of each modality data, ensuring that the final comprehensive data is more accurate and comprehensive.

[0116] S6, processing the fused comprehensive data to generate integrated data;

[0117] In this implementation, S6 specifically includes:

[0118] S61, the integrated data after fusion De-noising is performed, and median filtering is used to remove high-frequency noise and retain valid signals. The denoised data is expressed as ;

[0119] S62. De-noised data Normalize the feature so that the value of each feature is within a uniform scale: ;

[0120] in, The denoised data Middle The sample in The dimension value of the feature, For the The mean of the features, For the The standard deviation of the feature, For the The weighting coefficient of the feature, For the The variance of the feature, is the normalized data value;

[0121] S63, performing feature selection on the normalized data values, and selecting a strongly correlated feature subset that is highly correlated with the final task;

[0122] S64, based on the selected strongly correlated feature subset Reconstruct the normalized data values ​​and generate integrated data by calculating the weighted average of each feature : ;

[0123] in, For the The selected features are in the strongly correlated feature subset The weight in For the The selected features are in the normalized data The value in is the number of features in the strongly correlated feature subset.

[0124] This implementation method generates integrated data by processing the fused comprehensive data, thereby effectively improving the quality and availability of the data. Through feature extraction, dimensionality reduction and outlier detection in the processing process, it is possible to remove redundant information, eliminate fluctuations in the data, and optimize the structure of the data, making the integrated data more stable and reliable.

[0125] S7. During the operation of the intelligent system, based on the integrated data and new data source input, the association rules are updated in real time to generate updated association rules, and the fusion strategy is readjusted according to the updated association rules.

[0126] In this implementation, S7 specifically includes:

[0127] S71. During the operation of the intelligent system, new data source input is received , the new data source input includes new modality data and its related features;

[0128] S72. Input based on new data source , for existing integrated data Perform preliminary fusion processing to generate temporary fusion data : ;

[0129] in, For independent modality datasets In the new data source The weight in For the new data source data of each modality, For independent modality datasets in existing integrated data The weight of To integrate data Middle data of each modality, is the number of modes in the new data source, is the number of modes in the existing integrated data;

[0130] S73, based on the temporary fusion data , adaptively update the existing association rules, use incremental learning algorithms or other applicable methods to update the current association rule set Make real-time corrections to generate updated association rule sets : ;

[0131] in, Based on temporary fusion data and new data sources Association rules The function to update. is the updated association rule;

[0132] S74: According to the updated association rule set , recalculate the weight value of the modal data ;

[0133] S75: Based on the updated weight value , readjust the fusion strategy and generate a new fusion strategy .

[0134] This implementation enhances the dynamic adaptability of the system by updating the association rules in real time based on the integrated data and new data source input during the operation of the intelligent system, and readjusting the fusion strategy according to the updated association rules.

[0135] Example:

[0136] In order to verify the feasibility of the present invention, we applied it to the production monitoring system of a large intelligent manufacturing enterprise. Since the production line equipment of the enterprise involves multiple modal data sources, including sensor data, machine vision data and manual operation data, how to effectively integrate data of different modalities and improve the system's automated decision-making ability has become an important issue in improving production efficiency and reducing equipment failures. In response to this demand, the enterprise decided to adopt the intelligent system integration method based on multimodal data fusion of the present invention.

[0137] The system first obtains raw data from multiple data sources of different modalities, including physical data such as temperature, pressure, vibration from production line sensors, image data from machine vision systems, and manual operation records. After collection, these data are denoised and standardized through preprocessing steps to remove possible noise effects and unify the scale of each modality data to generate preprocessed data. This processing provides accurate and consistent input data for subsequent data fusion and analysis.

[0138] Next, based on the preprocessed data, the system uses an adaptive association rule mining algorithm to mine the potential associations between the modal data and generate a set of association rules between the modalities. By adaptively adjusting the association rules, the system can flexibly deal with the complex relationships between different modal data, avoiding the limitations of fixed rules in traditional methods. Subsequently, based on the generated association rules, the system dynamically adjusts the data fusion strategy, generates an adjusted fusion strategy, and performs weighted fusion on the modal data based on the strategy to generate fused comprehensive data.

[0139] By further processing the integrated data, the system generates the final integrated data, which not only contains the comprehensive information of each modal data, but also removes the redundant and unnecessary parts. After further analysis, the integrated data ultimately supports the intelligent decision-making of the production line and improves the accuracy and efficiency of equipment maintenance.

[0140] In order to verify the effectiveness of the present invention, the company collected relevant data within three months after applying the method. The results showed that the accuracy of fault prediction of production equipment was significantly improved through integrated data analysis. The specific data is shown in the following table:

[0141] Table 1: Operation effect data of production monitoring system of intelligent manufacturing enterprises

[0142] As can be seen from Table 1, within three months after the application of the present invention, the fault prediction accuracy of the intelligent manufacturing enterprise increased from 72.4% to 91.6%, the equipment maintenance cost was significantly reduced by about 35%, and the equipment downtime due to failure was reduced by 5 hours / month. In addition, the data processing time was reduced from 45 minutes per batch to 20 minutes, and the system response time was also shortened from 1 hour to 15 minutes, which greatly improved the work efficiency of the production line. The production efficiency increased by about 42%, and the monthly output increased from 5,500 pieces to 7,800 pieces.

[0143] From these data, it can be seen that the present invention effectively improves the accuracy and efficiency of multimodal data fusion by adaptively mining association rules and dynamically adjusting the fusion strategy. When dealing with a dynamically changing production environment, the system can update and adjust the data fusion strategy in real time, optimize the maintenance and fault prediction of production line equipment, thereby significantly improving production efficiency and equipment utilization, and reducing operating costs.

[0144] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent system integration method based on multimodal data fusion, characterized in that: The steps include: S1. Obtain multiple data sources of different modalities; S2, preprocessing the raw data from multiple data sources of different modalities, including denoising and standardization, to generate preprocessed data; S3. Based on the preprocessed data, an adaptive association rule mining algorithm is used to mine the potential association relationship between the preprocessed data of each modality and generate an inter-modality association rule set; S4, dynamically adjusting the data fusion strategy according to the set of association rules between modalities, and generating an adjusted fusion strategy; S5. According to the adjusted fusion strategy, weighted fusion is performed on the pre-processed data of each modality to generate fused comprehensive data; S6, processing the fused comprehensive data to generate integrated data; S7. During the operation of the intelligent system, based on the integrated data and new data source input, the set of inter-modality association rules is updated in real time to generate updated association rules, and the fusion strategy is readjusted according to the updated association rules.

2. The intelligent system integration method based on multimodal data fusion according to claim 1 is characterized in that: The S3 specifically includes: S31, performing modal division on the preprocessed data to obtain independent modal data sets for each modal data, wherein the preprocessed data is represented as ,in Indicates independent modal data sets, n represents the total number of independent modal data in the modal data set, including All sample data in the independent modal data set, each sample data consists of a feature vector Composition, of which is the number of eigenvectors, and Indicates In the modal The value of the eigenvector; S32, in each modality data set, select an appropriate subset of feature vectors, and calculate the correlation between the feature vectors in the selected feature vector subset, the correlation To reflect the characteristics and Features The relationship between: ； in, and Respectively represent Features and The feature in The values ​​in the subset of eigenvectors, and Respectively Features and Features in the modal The mean value under is the number of eigenvector subsets; S33, according to the correlation between the feature vectors, an adaptive association rule mining algorithm is used to set the support threshold and confidence threshold , mining the potential association rules between modal data and generating the association rule set between modalities , where each association rule Expressed as ,in and are two feature sets and satisfy the following conditions: ； in, for and The probability of simultaneous occurrence, For Derived The conditional probability of S34, according to the generated set of association rules between modalities, by calculating the lifting degree of each rule To evaluate the strength of each association rule, we can get a set of association rules with higher strength: ； in, and The feature sets and The marginal probability of , then it means and There is a strong positive correlation between them; S35: Filter out association rules with stronger inter-modal dependencies from the association rule set with higher strength, and generate a final inter-modal association rule set. , where each association rule Indicates the relationship between modes.

3. The intelligent system integration method based on multimodal data fusion according to claim 2 is characterized in that: The S4 specifically includes: S41, based on the inter-modality association rule set, calculate each association rule The corresponding weight value , where each weight value Reflects the contribution of the association rule to the modal data fusion process: ； in, Association rules The degree of improvement, Association rules middle and The probability of simultaneous occurrence, and are other feature sets in the inter-modal association rule set, For the comprehensive contribution of all association rules, is the number of rules in the inter-modality association rule set; S42. According to each association rule The weight value , calculate each modal data set Weights in data fusion : ； in, For independent modality datasets The correlation value between the modalities in the association rules, is the weight value of the association rule, and n represents the total number of independent modal data in the modal data set; S43. For each independent modality data set Weight Perform normalization to obtain the normalized weight value ; S44, according to the normalized weight value , generate the adjusted fusion strategy; this strategy uses the weighted average method to calculate the normalized weight value of each modal data. Perform weighted fusion on modal data to obtain the adjusted fusion strategy : ； Where n represents the total number of independent modal data in the modal data set; S45. According to the adjusted fusion strategy, feature selection and dimensionality reduction processing of the modal data are performed, and the normalized weight value of each independent modal data set is adjusted in real time.

4. The intelligent system integration method based on multimodal data fusion according to claim 3 is characterized in that: The S5 specifically includes: S51, based on the adjusted fusion strategy , for each modality dataset According to its normalized weight value Perform weighting to generate a weighted modal data set : ； in, Indicates An independent modality dataset; S52, for all weighted modal data Add and generate integrated data : ； in, is the integrated data after fusion. is the weighted modal data set, n represents the total number of independent modal data in the modal data set; S53. Comprehensive data after fusion Perform feature extraction and standardization to generate processed fusion data ; S54, based on the processed fusion data , optimize the weight of each modal data in the fusion process, and adjust the fusion strategy in real time by evaluating the contribution of each modal data set; S55: regenerate the weighted modality data set and update the comprehensive data according to the adjusted fusion strategy .

5. The intelligent system integration method based on multimodal data fusion according to claim 4 is characterized in that: The S53 specifically includes: S531. Comprehensive data after fusion Perform feature extraction, select feature subsets, and measure the contribution of each feature to the feature subset by calculating the importance of the features contained in the feature subset: ； in, For the Features The importance value of Features The variance of is the total number of features, Features The variance of S532, based on the selected feature subset, the fused comprehensive data Standardization is performed so that the mean of each feature is , the standard deviation is , get the standardized data value ; S533, data value after standardization The principal component analysis is performed on the dimensionality reduction operation. By calculating the characteristic covariance matrix and solving its eigenvalues ​​and eigenvectors, the principal components with the largest variance are selected and the data are mapped to these principal components to generate the data representation after dimensionality reduction. ; S534, according to the data after dimension reduction , perform outlier detection on the reduced-dimensional data representation, remove outliers, and generate processed fused data .

6. The intelligent system integration method based on multimodal data fusion according to claim 1, characterized in that: The S6 specifically includes: S61, the integrated data after fusion De-noising is performed, and median filtering is used to remove high-frequency noise and retain valid signals. The denoised data is expressed as ; S62. De-noised data Normalize the feature so that the value of each feature is within a uniform scale: ； in, The denoised data Middle The sample in The dimension value of the feature, For the The mean of the features, For the The standard deviation of the feature, For the The weighting coefficient of the feature, For the The variance of the feature, is the normalized data value; S63, performing feature selection on the normalized data values, and selecting a strongly correlated feature subset that is highly correlated with the final task; S64, based on the selected strongly correlated feature subset Reconstruct the normalized data values ​​and generate integrated data by calculating the weighted average of each feature : ； in, For the The selected features are in the strongly correlated feature subset The weight in For the The selected features are in the normalized data The value in is the number of features in the strongly correlated feature subset.

7. The intelligent system integration method based on multimodal data fusion according to claim 1, characterized in that: The S7 specifically includes: S71. During the operation of the intelligent system, new data source input is received , the new data source input includes new modality data and its related features; S72. Input based on new data source , for existing integrated data Perform preliminary fusion processing to generate temporary fusion data : ； in, For independent modality datasets In the new data source The weight in For the new data source data of each modality, For independent modality datasets in existing integrated data The weight of To integrate data Middle data of each modality, is the number of modes in the new data source, is the number of modes in the existing integrated data; S73, based on the temporary fusion data , adaptively update the existing association rules, use incremental learning algorithms or other applicable methods to update the current association rule set Make real-time corrections to generate updated association rule sets : ； in, Based on temporary fusion data and new data sources Association rules The function to update. is the updated association rule; S74: According to the updated association rule set , recalculate the weight value of the modal data ; S75, according to the updated weight value , readjust the fusion strategy and generate a new fusion strategy .

Images