Intelligent System Integration Method Based on Multimodal Data Fusion

Abstract

The present invention discloses an intelligent system integration method based on multi-modal data fusion, comprising: S1, obtaining multiple data sources of different modalities; S2, preprocessing the original data in the multiple data sources of different modalities to generate preprocessed data; S3, based on the preprocessed data, mining the potential correlation relationships between the data of each modality to generate a set of inter-modal association rules; S4, dynamically adjusting the data fusion strategy according to the set of inter-modal association rules to generate an adjusted fusion strategy; S5, performing weighted fusion on the data of each modality according to the adjusted fusion strategy to generate fused comprehensive data; S6, processing the fused comprehensive data to generate integrated data; S7, during the operation of the intelligent system, based on the integrated data and new data source input, updating the association rules in real time. The present invention has the advantages of strong flexibility, high efficiency, strong adaptability and low data processing cost.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent systems, and particularly to an intelligent system integration method based on multi-modal data fusion. Background Art

[0002] With the continuous progress of information technology and the wide application of artificial intelligence, intelligent systems have been widely applied and promoted in multiple fields. Especially in multi-modal data fusion, how to effectively integrate data from different sources and different types to improve the comprehensive decision-making ability of the system has become an important research topic in the current research and development of intelligent systems. Multi-modal data fusion can provide a more comprehensive perspective and decision support by integrating information from multiple data sources such as vision, voice, and sensors. However, existing intelligent system integration methods face multiple technical challenges.

[0003] In the prior art, traditional multi-modal data fusion methods mostly rely on static fusion rules and fixed data processing flows. These methods usually use deep learning models or predefined rules to fuse data, but when facing the complex associations between different modal data, they often cannot flexibly adapt and generate the optimal fusion strategy. Specifically, the prior art has the following deficiencies:

[0004] 1. Lack of dynamic adaptability: Existing fusion methods usually adopt fixed models or rules and cannot automatically adjust the fusion strategy in real time according to new data sources or data changes, resulting in poor performance when dealing with new and variable data sources.

[0005] 2. Information redundancy and conflict problems: Traditional methods often cannot effectively identify and handle the redundancy and conflict between modal data when processing multi-modal data, which easily leads to inaccuracy or inconsistency in information fusion, thus affecting the final system decision-making.

[0006] 3. High dependence on labeled data: Most existing intelligent system integration methods rely on a large amount of labeled data for training, which seriously restricts the data annotation cost and the availability of labeled data in practical applications. Especially in scenarios where data changes rapidly or diversely, traditional methods are difficult to provide sufficient flexibility.

[0007] 4. Fixity of fusion strategy: The data fusion strategy in traditional methods is usually predefined by the deep learning network structure, lacking the ability to adaptively adjust to different data sources and unable to dynamically optimize the fusion mode according to the associations between different modal data.

[0008] Therefore, how to provide an intelligent system integration method based on multi-modal data fusion is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0009] An object of the present invention is to provide an intelligent system integration method based on multi-modal data fusion. The present invention makes full use of the adaptive association rule mining technology and the inter-modal association network framework, and details how to optimize the data fusion process by dynamically adjusting the fusion strategy, reduce information redundancy and conflicts, reduce the dependence on a large amount of labeled data, and has the advantages of strong flexibility, high efficiency, strong adaptability, and low data processing cost.

[0010] The intelligent system integration method based on multi-modal data fusion according to an embodiment of the present invention includes the following steps:

[0011] S1. Obtain multiple data sources of different modalities;

[0012] S2. Preprocess the raw data in the multiple data sources of different modalities, including denoising and standardization processing, to generate preprocessed data;

[0013] S3. Based on the preprocessed data, use an adaptive association rule mining algorithm to mine the potential association relationships between the preprocessed data of each modality, and generate an inter-modal association rule set;

[0014] S4. According to the inter-modal association rule set, dynamically adjust the data fusion strategy to generate an adjusted fusion strategy;

[0015] S5. According to the adjusted fusion strategy, perform weighted fusion on the preprocessed data of each modality to generate fused comprehensive data;

[0016] S6. Process the fused comprehensive data to generate integrated data;

[0017] S7. During the operation of the intelligent system, based on the integrated data and new data source inputs, update the inter-modal association rule set in real time to generate updated association rules, and re-adjust the fusion strategy according to the updated association rules.

[0018] Optionally, the S3 specifically includes:

[0019] S31. Perform modality partitioning on the preprocessed data to obtain independent modality data sets for each modality data. The preprocessed data is represented as , where represents the th independent modality data set, n represents the total number of independent modality data in the modality data set, includes all sample data in the th independent modality data set, and each sample data is composed of a feature vector , where is the number of feature vectors, and Indicates the value of the th eigenvector in the th mode;

[0020] S32. In each modal dataset, select an appropriate subset of eigenvectors and calculate the degree of association between the eigenvectors in the selected subset of eigenvectors. The degree of association is used to reflect the relationship between features :

[0021] ;

[0022] where and respectively represent the values of the th feature and the th feature in the th subset of eigenvectors, and are respectively the means of the th feature and the th feature in mode , is the number of subsets of eigenvectors;

[0023] S33. According to the degree of association between the eigenvectors, use an adaptive association rule mining algorithm to mine the potential association rules between modal data by setting a support threshold and a confidence threshold , and generate an association rule set between modes. Each association rule is expressed as , where and are two feature sets and satisfy the following conditions:

[0024] ;

[0025] where is the probability that and occur simultaneously, is the conditional probability of deriving from to ;

[0026] S34. According to the generated association rule set between modes, evaluate the strength of each association rule by calculating the lift of each rule, and obtain an association rule set with higher strength:

[0027] ;

[0028] Among them, and are the marginal probabilities of the feature sets and respectively. If the lift , it means that and have a strong positive correlation;

[0029] S35. From the association rule set with higher strength, screen out the association rules with strong inter-modal dependencies to generate the final inter-modal association rule set , where each association rule represents the inter-modal association relationship.

[0030] Optionally, the S4 specifically includes:

[0031] S41. Based on the inter-modal association rule set, calculate the weight value corresponding to each association rule , where each weight value reflects the contribution degree of this association rule to the modal data fusion process:

[0032] ;

[0033] Among them, is the lift of the association rule , is the probability that and and occur simultaneously in the association rule and are other feature sets in the inter-modal association rule set, is the comprehensive contribution of all association rules, is the number of rules in the inter-modal association rule set;

[0034] S42. According to the weight value of each association rule , calculate the weight of each modal data set in the data fusion:

[0035] ;

[0036] Among them, is the correlation value between the independent modal data set and the modal in the association rule, is the weight value of the association rule, and n represents the total number of independent modal data in the modal data set;

[0037] S43. For each independent modal dataset weight is normalized to obtain the normalized weight value ;

[0038] S44. According to the normalized weight value , an adjusted fusion strategy is generated; this strategy, through weighted averaging, weights and fuses the modal data according to the normalized weight values of each modal data to obtain the adjusted fusion strategy :

[0039] ;

[0040] where n represents the total number of independent modal data in the modal dataset;

[0041] S45. According to the adjusted fusion strategy, feature selection and dimensionality reduction processing are performed on the modal data, and the normalized weight values of each independent modal dataset are adjusted in real time.

[0042] Optionally, the specific steps of S5 include:

[0043] S51. Based on the adjusted fusion strategy , each modal dataset is weighted according to its normalized weight value to generate the weighted modal dataset :

[0044] ;

[0045] where represents the th independent modal dataset;

[0046] S52. The sum of all weighted modal data is calculated to generate the integrated data after fusion :

[0047] ;

[0048] where is the integrated data after fusion, is the weighted modal dataset, and n represents the total number of independent modal data in the modal dataset;

[0049] S53. Feature extraction and standardization processing are performed on the integrated data after fusion to generate the processed fusion data ;

[0050] S54. According to the processed fusion data , optimize the weights of each modality data during the fusion process, evaluate the contribution degree of each modality data set, and adjust the fusion strategy in real time;

[0051] S55. According to the adjusted fusion strategy, regenerate the weighted modality data set and update the comprehensive data .

[0052] Optionally, the S53 specifically includes:

[0053] S531. Extract features from the fused comprehensive data , select a feature subset, and measure the degree of contribution of each feature to the feature subset by calculating the importance of the features included in the feature subset:

[0054] ;

[0055] Among them, is the importance value of the th feature , is the variance of the feature , is the total number of features, is the variance of the feature ;

[0056] S532. According to the selected feature subset, perform standardization processing on the fused comprehensive data so that the mean of each feature is , and the standard deviation is , obtaining the standardized data value ;

[0057] S533. Perform principal component analysis dimensionality reduction operation on the standardized data value , calculate the feature covariance matrix, solve its eigenvalues and eigenvectors, select the principal components with the largest variance, and map the data to these principal components to generate the dimensionality-reduced data representation ;

[0058] S534. According to the dimensionality-reduced data representation , perform outlier detection on the dimensionality-reduced data representation, remove the outliers, and generate the processed fusion data .

[0059] Optionally, the S6 specifically includes:

[0060] S61. For the fused comprehensive data Perform denoising processing, use median filtering to remove high-frequency noise, and retain the effective signal. The data after denoising is represented as ;

[0061] S62. For the data after denoising Perform normalization processing to make the values of each feature within a unified scale range:

[0062] ;

[0063] Among them, is the value of the th sample in the data after denoising in the dimension of the th feature, is the mean of the th feature, is the standard deviation of the th feature, is the weighting coefficient of the th feature, is the variance of the th feature, is the value of the data after normalization;

[0064] S63. Perform feature selection on the values of the data after normalization, and select a subset of strongly correlated features that are highly relevant to the final task;

[0065] S64. According to the selected subset of strongly correlated features Reconstruct the values of the data after normalization, and calculate by weighted averaging of each feature to generate integrated data :

[0066] ;

[0067] Among them, is the weight of the th selected feature in the subset of strongly correlated features , is the value of the th selected feature in the data after normalization , is the number of features in the subset of strongly correlated features.

[0068] Optionally, the specific steps of S7 include:

[0069] S71. During the operation of the intelligent system, receive a new data source input , and the new data source input includes new modal data and its related features;

[0070] S72. According to the new data source input , perform preliminary fusion processing on the existing integrated data to generate temporary fusion data :

[0071] ;

[0072] Among them, is the weight of the independent modal data set in the new data source , is the data of the th modality in the new data source, is the weight of the independent modal data set in the existing integrated data, is the th modality data in the integrated data, is the number of modalities in the new data source, is the number of modalities in the existing integrated data;

[0073] S73. Based on the temporary fusion data , adaptively update the existing association rules, and use an incremental learning algorithm or other applicable methods to perform real-time correction on the current association rule set to generate an updated association rule set

[0074] ;

[0075] Among them, is the function to update the association rule according to the temporary fusion data and the new data source , is the updated association rule;

[0076] S74. According to the updated association rule set , recalculate the weight values of the modal data ;

[0077] S75. According to the updated weight values , readjust the fusion strategy to generate a new fusion strategy .

[0078] The beneficial effects of the present invention are:

[0079] (1) By adopting the adaptive association rule mining technology and the inter-modal association network framework, the present invention dynamically adjusts the data fusion strategy, thus effectively solving the problems of information redundancy, information conflict and fixed fusion strategy in traditional multi-modal data fusion methods. This method automatically optimizes the fusion strategy according to the potential relevance between modal data, improving the flexibility and adaptability of the system in the face of complex and changing data sources.

[0080] (2) By reducing the dependence on a large amount of labeled data and utilizing the adaptive data fusion strategy and the dynamic relevance between modalities, the present invention significantly reduces the cost and workload of data annotation, enabling the system to operate efficiently in an environment where data sources change rapidly, and is particularly suitable for processing large-scale and heterogeneous data sources.

[0081] (3) Through the adaptive association rule update mechanism, the present invention can adjust the fusion strategy in real time according to the input of new data sources and optimize the weighted fusion process of modal data, thus ensuring that the system can continuously adapt to the changes of different data sources, and improving the real-time performance, accuracy and stability of intelligent system integration. This dynamic adaptive mechanism significantly improves the system's ability to handle different modal data inputs, especially showing excellent effects in the fields of intelligent decision support and multi-modal data analysis. Brief Description of the Drawings

[0082] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, but do not constitute a limitation to the present invention. In the drawings:

[0083] Figure 1 is a flowchart of the intelligent system integration method based on multi-modal data fusion proposed by the present invention. Detailed Embodiments

[0084] Now, the present invention will be further described in detail with reference to the drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic way, so they only show the components related to the present invention.

[0085] Reference Figure 1 , the intelligent system integration method based on multi-modal data fusion includes the following steps:

[0086] S1. Obtain multiple data sources of different modalities;

[0087] In this embodiment, by obtaining multiple data sources of different modalities, a comprehensive integration of data is provided, enabling the system to obtain information from multiple dimensions. This process not only enriches the data sources but also improves the diversity and comprehensiveness of the data, thereby enhancing the processing ability of the intelligent system for diverse data in complex environments.

[0088] S2. Preprocess the raw data in multiple data sources of different modalities, remove noise and perform normalization to generate preprocessed data;

[0089] In this embodiment, by preprocessing the raw data in multiple data sources of different modalities, removing noise and performing normalization, the quality and consistency of the data are significantly improved. Denoising and normalization ensure the comparability of data in each modality in the same data space, thus avoiding errors caused by noise interference and inconsistent data scales.

[0090] S3. Based on the preprocessed data, use the adaptive association rule mining algorithm to mine the potential association relationships between data in each modality and generate a set of inter-modal association rules;

[0091] In this embodiment, S3 specifically includes:

[0092] S31. Perform modality partitioning on the preprocessed data to obtain independent modality data sets for each modality data. The preprocessed data is represented as , where represents the th independent modality data set, n represents the total number of independent modality data in the modality data set, including all sample data in the th independent modality data set. Each sample data consists of a feature vector composed of, where is the number of feature vectors, and represents the value of the th feature vector in the th modality;

[0093] S32. In each modality data set, select an appropriate subset of feature vectors and calculate the degree of association between the feature vectors in the selected subset of feature vectors. The degree of association is used to reflect the mutual relationship between feature and feature :

[0094] ;

[0095] Among them, and respectively represent the values of the th feature and the th feature in the th subset of feature vectors, and are respectively the means of the th feature and the th feature in modality , is the number of subsets of feature vectors;

[0096] S33. According to the correlation degrees between the feature vectors, adopt an adaptive association rule mining algorithm, and by setting a support threshold and a confidence threshold , mine the potential association rules between the modal data, and generate an association rule set between the modes , where each association rule is expressed as , where and are two feature sets and satisfy the following conditions:

[0097] ;

[0098] Among them, is and the probability of simultaneous occurrence, is the conditional probability of deriving from to ;

[0099] S34. According to the generated association rule set between the modes, evaluate the strength of each rule by calculating the lift of each rule, and obtain a set of association rules with higher strength:

[0100] ;

[0101] Among them, and are the marginal probabilities of the feature sets and respectively. If the lift , it means that there is a strong positive correlation between and ;

[0102] S35. From the set of association rules with higher strength, screen out the association rules with stronger dependence between the modes, and generate a final set of association rules between the modes , where each association rule represents the association relationship between the modes.

[0103] In this embodiment, based on the preprocessed data, an adaptive association rule mining algorithm is adopted to mine the potential association relationships between the modal data, and an association rule set between the modes is generated. This process can automatically discover the deep-seated associations between different modal data, avoiding the limitations and fixities of manually setting rules.

[0104] S4. Dynamically adjust the data fusion strategy according to the set of inter-modal association rules to generate an adjusted fusion strategy;

[0105] In this embodiment, S4 specifically includes:

[0106] S41. Based on the set of inter-modal association rules, calculate the weight value corresponding to each association rule , where each weight value reflects the contribution degree of this association rule to the modal data fusion process:

[0107] ;

[0108] Among them, is the lift of the association rule , is the probability that in the association rule and occur simultaneously, and are other feature sets in the set of inter-modal association rules, is the comprehensive contribution of all association rules, is the number of rules in the set of inter-modal association rules;

[0109] S42. According to the weight value of each association rule , calculate the weight of each modal data set in data fusion:

[0110] ;

[0111] Among them, is the correlation value between the independent modal data set and the modal in the association rule, is the weight value of the association rule, and n represents the total number of independent modal data in the modal data set;

[0112] S43. Normalize the weight of each independent modal data set to obtain a normalized weight value ;

[0113] S44. Generate an adjusted fusion strategy according to the normalized weight value ; this strategy, through the method of weighted average, performs weighted fusion on the modal data according to the normalized weight value of each modal data to obtain an adjusted fusion strategy :

[0114] ;

[0115] Among them, n represents the total number of independent modal data in the modal dataset;

[0116] S45. According to the adjusted fusion strategy, perform feature selection and dimensionality reduction processing on the modal data, and adjust the normalized weight value of each independent modal dataset in real time.

[0117] In this embodiment, by dynamically adjusting the data fusion strategy according to the set of inter-modal association rules to generate an adjusted fusion strategy, the flexible adaptation to the relationship between different modal data is achieved. This method can optimize the data fusion strategy in real time according to the association rules, avoid the limitations of the fixed fusion mode, and improve the adaptability and intelligence level of the system when processing complex data sources.

[0118] S5. According to the adjusted fusion strategy, perform weighted fusion on each modal data to generate the fused comprehensive data;

[0119] In this embodiment, S5 specifically includes:

[0120] S51. Based on the adjusted fusion strategy , for each modal dataset According to its normalized weight value Perform weighting to generate the weighted modal dataset :

[0121] ;

[0122] Among them, represents the th independent modal dataset;

[0123] S52. Sum all the weighted modal data to generate the fused comprehensive data :

[0124] ;

[0125] Among them, is the fused comprehensive data, is the weighted modal dataset, and n represents the total number of independent modal data in the modal dataset;

[0126] S53. Perform feature extraction and normalization processing on the fused comprehensive data to generate the processed fused data ;

[0127] S54. According to the processed fusion data , optimize the weights of each modality data during the fusion process, evaluate the contribution degree of each modality dataset, and adjust the fusion strategy in real time;

[0128] S55. According to the adjusted fusion strategy, regenerate the weighted modality dataset and update the comprehensive data .

[0129] The specific steps of S53 include:

[0130] S531. Extract features from the fused comprehensive data , select a feature subset, and measure the degree of contribution of each feature to the feature subset by calculating the importance of the features included in the feature subset:

[0131] ;

[0132] Among them, is the importance value of the th feature , is the variance of the feature , is the total number of features, is the variance of the feature ;

[0133] S532. According to the selected feature subset, perform standardization processing on the fused comprehensive data so that the mean of each feature is , and the standard deviation is , obtaining the standardized data value ;

[0134] S533. Perform principal component analysis dimensionality reduction operation on the standardized data value . By calculating the feature covariance matrix, solving its eigenvalues and eigenvectors, select the principal components with the largest variance, and map the data to these principal components to generate the dimensionality-reduced data representation ;

[0135] S534. According to the dimensionality-reduced data representation , perform outlier detection on the dimensionality-reduced data representation, remove outliers, and generate the processed fusion data .

[0136] In this embodiment, by performing weighted fusion on each modality data according to the adjusted fusion strategy, integrated data after fusion is generated, thereby realizing the effective integration of each modality data. Through weighted fusion, the system can dynamically adjust the contribution degree of each modality according to the relative importance of each modality data, ensuring that the final integrated data is more accurate and comprehensive.

[0137] S6. Process the integrated data after fusion to generate integrated data;

[0138] In this embodiment, S6 specifically includes:

[0139] S61. Process the integrated data after fusion to perform denoising processing, using median filtering to remove high-frequency noise and retain the effective signal. The data after denoising is denoted as ;

[0140] S62. Perform normalization processing on the data after denoising to make the value of each feature within a unified scale range:

[0141] ;

[0142] where, is the dimensional value of the th sample in the th feature of the data after denoising , is the mean value of the th feature, is the standard deviation of the th feature, is the weighted coefficient of the th feature, is the variance of the th feature, is the data value after normalization;

[0143] S63. Perform feature selection on the data value after normalization to select a subset of strongly correlated features that are highly relevant to the final task;

[0144] S64. Reconstruct the data value after normalization according to the selected subset of strongly correlated features by calculating the weighted average of each feature to generate integrated data :

[0145] ;

[0146] where, is the weight of the th selected feature in the subset of strongly correlated features , is the value of the th selected feature in the normalized data . is the number of features in the strongly correlated feature subset.

[0147] In this embodiment, by processing the fused comprehensive data, integrated data is generated, thereby effectively improving the quality and usability of the data. Through feature extraction, dimensionality reduction, and outlier detection in the processing process, redundant information can be removed, fluctuations in the data can be eliminated, and the structure of the data can be optimized, making the integrated data more stable and reliable.

[0148] S7. During the operation of the intelligent system, based on the integrated data and the input of new data sources, the association rules are updated in real time to generate updated association rules, and the fusion strategy is readjusted according to the updated association rules.

[0149] In this embodiment, S7 specifically includes:

[0150] S71. During the operation of the intelligent system, receive the input of new data sources , where the input of the new data source includes new modal data and its related features;

[0151] S72. According to the input of the new data source , perform preliminary fusion processing on the existing integrated data to generate temporary fusion data :

[0152] ;

[0153] where is the weight of the independent modal data set in the new data source , is the th modal data in the new data source, is the weight of the independent modal data set in the existing integrated data, is the integrated data the th modal data, is the number of modalities in the new data source, is the number of modalities in the existing integrated data;

[0154] S73. Based on the temporary fusion data , adaptively update the existing association rules, and use an incremental learning algorithm or other applicable methods to perform real-time correction on the current association rule set to generate an updated association rule set :

[0155] ;

[0156] wherein, is a function for updating the association rules based on the temporary fusion data and the new data source, and is the updated association rule;

[0157] S74. Recalculate the weight values of the modal data according to the updated set of association rules;

[0158] ; S75. Readjust the fusion strategy according to the updated weight values to generate a new fusion strategy.

[0159] .

[0160] In this embodiment, during the operation of the intelligent system, based on the integrated data and the input of the new data source, the association rules are updated in real time, and the fusion strategy is readjusted according to the updated association rules, thereby enhancing the dynamic adaptability of the system.

[0161] Example:

[0162] To verify the feasibility of the present invention, we applied it to the production monitoring system of a large-scale intelligent manufacturing enterprise. Since the production line equipment of this enterprise involves multiple modal data sources, covering sensor data, machine vision data, and manual operation data, how to effectively integrate different modal data and improve the automated decision-making ability of the system has become an important issue for improving production efficiency and reducing equipment failures. In response to this demand, the enterprise decided to adopt the intelligent system integration method based on multi-modal data fusion of the present invention.

[0163] ​​​Next, based on the preprocessed data, the system uses an adaptive association rule mining algorithm to discover the potential association relationships between multi-modal data, generating a set of association rules between modalities. By adaptively adjusting the association rules, the system can flexibly handle the complex relationships between different modal data, avoiding the limitations of fixed rules in traditional methods. Subsequently, according to the generated association rules, the system dynamically adjusts the data fusion strategy, generates an adjusted fusion strategy, and performs weighted fusion on the multi-modal data based on this strategy to generate the fused comprehensive data.

[0164] By further processing the fused comprehensive data, the system generates the final integrated data, which not only contains the comprehensive information of each modal data but also removes redundant and unnecessary parts. After further analysis of the integrated data, it ultimately supports the intelligent decision-making of the production line, improving the accuracy and efficiency of equipment maintenance.

[0165] To verify the effectiveness of the present invention, the enterprise collected relevant data within three months after applying this method. The results showed that the accuracy of fault prediction for production equipment was significantly improved through integrated data analysis. The specific data is shown in the following table:

[0166] Table 1: Data Table of the Operation Effect of the Production Monitoring System of an Intelligent Manufacturing Enterprise

[0167]

[0168] As can be seen from Table 1, within three months after applying the present invention, the fault prediction accuracy rate of the intelligent manufacturing enterprise increased from 72.4% to 91.6%, the equipment maintenance cost was significantly reduced by approximately 35%, and the equipment downtime due to faults decreased by 5 hours per month. In addition, the data processing time was reduced from 45 minutes per batch to 20 minutes, and the system response time was also shortened from 1 hour to 15 minutes, which greatly improved the working efficiency of the production line. The production efficiency increased by approximately 42%, and the monthly output increased from 5,500 pieces to 7,800 pieces.

[0169] From these data, it can be seen that the present invention effectively improves the accuracy and efficiency of multi-modal data fusion by means of adaptive association rule mining and dynamic adjustment of the fusion strategy. When dealing with a dynamically changing production environment, the system can update and adjust the data fusion strategy in real time, optimize the maintenance and fault prediction of production line equipment, thereby significantly improving production efficiency and equipment utilization rate and reducing operating costs.

[0170] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes should be covered within the protection scope of the present invention.

Claims

1. An intelligent system integration method based on multimodal data fusion, characterized in that: The steps include: S1. Obtain multiple data sources of different modalities; S2, preprocessing the raw data from multiple data sources of different modalities, including denoising and standardization, to generate preprocessed data; S3. Based on the preprocessed data, an adaptive association rule mining algorithm is used to mine the potential association relationship between the preprocessed data of each modality and generate an inter-modality association rule set; S4, dynamically adjusting the data fusion strategy according to the set of association rules between modalities, and generating an adjusted fusion strategy; S5. According to the adjusted fusion strategy, weighted fusion is performed on the pre-processed data of each modality to generate fused comprehensive data; S6, processing the fused comprehensive data to generate integrated data; S7. During the operation of the intelligent system, based on the integrated data and new data source input, the set of inter-modality association rules is updated in real time to generate updated association rules, and the fusion strategy is readjusted according to the updated association rules; The S3 specifically includes: S31, performing modal division on the preprocessed data to obtain independent modal data sets for each modal data, wherein the preprocessed data is represented as ,in Indicates independent modal data sets, n represents the total number of independent modal data in the modal data set, including All sample data in the independent modal data set, each sample data consists of a feature vector Composition, of which is the number of eigenvectors, and Indicates In the modal The value of the eigenvector; S32, in each modality data set, select an appropriate subset of feature vectors, and calculate the correlation between the feature vectors in the selected feature vector subset, the correlation To reflect the characteristics and Features The relationship between: ； in, and Respectively represent Features and The feature in The values ​​in the subset of eigenvectors, and Respectively Features and Features in the modal The mean value under is the number of eigenvector subsets; S33, according to the correlation between the feature vectors, an adaptive association rule mining algorithm is used to set the support threshold and confidence threshold , mining the potential association rules between modal data and generating the association rule set between modalities , where each association rule Expressed as ,in and are two feature sets and satisfy the following conditions: ； in, for and The probability of simultaneous occurrence, For Derived The conditional probability of S34, according to the generated set of association rules between modalities, by calculating the lifting degree of each rule To evaluate the strength of each association rule, we can get a set of association rules with higher strength: ； in, and The feature sets and The marginal probability of , then it means and There is a strong positive correlation between them; S35: Filter out association rules with stronger inter-modal dependencies from the association rule set with higher strength, and generate a final inter-modal association rule set. , where each association rule Indicates the relationship between modes; The S4 specifically includes: S41, based on the inter-modality association rule set, calculate each association rule The corresponding weight value , where each weight value Reflects the contribution of the association rule to the modal data fusion process: ； in, Association rules The degree of improvement, Association rules middle and The probability of simultaneous occurrence, and are other feature sets in the inter-modal association rule set, For the comprehensive contribution of all association rules, is the number of rules in the inter-modality association rule set; S42. According to each association rule The weight value , calculate each modal data set Weights in data fusion : ； in, For independent modality datasets The correlation value between the modalities in the association rules, is the weight value of the association rule, and n represents the total number of independent modal data in the modal data set; S43. For each independent modality data set Weight Perform normalization to obtain the normalized weight value ; S44, according to the normalized weight value , generate the adjusted fusion strategy; this strategy uses the weighted average method to calculate the normalized weight value of each modal data. Perform weighted fusion on modal data to obtain the adjusted fusion strategy : ； Where n represents the total number of independent modal data in the modal data set; S45. According to the adjusted fusion strategy, feature selection and dimensionality reduction processing of the modal data are performed, and the normalized weight value of each independent modal data set is adjusted in real time.

2. The intelligent system integration method based on multimodal data fusion according to claim 1 is characterized in that: The S5 specifically includes: S51, based on the adjusted fusion strategy , for each modality dataset According to its normalized weight value Perform weighting to generate a weighted modal data set : ； in, Indicates An independent modality dataset; S52, for all weighted modal data Add and generate integrated data : ； in, is the integrated data after fusion. is the weighted modal data set, n represents the total number of independent modal data in the modal data set; S53. Comprehensive data after fusion Perform feature extraction and standardization to generate processed fusion data ; S54, based on the processed fusion data , optimize the weight of each modal data in the fusion process, and adjust the fusion strategy in real time by evaluating the contribution of each modal data set; S55: regenerate the weighted modality data set and update the comprehensive data according to the adjusted fusion strategy .

3. The intelligent system integration method based on multimodal data fusion according to claim 2 is characterized in that: The S53 specifically includes: S531. Comprehensive data after fusion Perform feature extraction, select feature subsets, and measure the contribution of each feature to the feature subset by calculating the importance of the features contained in the feature subset: ； in, For the Features The importance value of Features The variance of is the total number of features, Features The variance of S532, based on the selected feature subset, the fused comprehensive data Standardization is performed so that the mean of each feature is , the standard deviation is , get the standardized data value ; S533, data value after standardization The principal component analysis is performed on the dimensionality reduction operation. By calculating the characteristic covariance matrix and solving its eigenvalues ​​and eigenvectors, the principal components with the largest variance are selected and the data are mapped to these principal components to generate the data representation after dimensionality reduction. ; S534, according to the data after dimension reduction , perform outlier detection on the reduced-dimensional data representation, remove outliers, and generate processed fused data .

4. The intelligent system integration method based on multimodal data fusion according to claim 1, characterized in that: The S6 specifically includes: S61, the integrated data after fusion De-noising is performed, and median filtering is used to remove high-frequency noise and retain valid signals. The denoised data is expressed as ; S62. De-noised data Normalize the feature so that the value of each feature is within a uniform scale: ； in, The denoised data Middle The sample in The dimension value of the feature, For the The mean of the features, For the The standard deviation of the feature, For the The weighting coefficient of the feature, For the The variance of the feature, is the normalized data value; S63, performing feature selection on the normalized data values, and selecting a strongly correlated feature subset that is highly correlated with the final task; S64, based on the selected strongly correlated feature subset Reconstruct the normalized data values ​​and generate integrated data by calculating the weighted average of each feature : ； in, For the The selected features are in the strongly correlated feature subset The weight in For the The selected features are in the normalized data The value in is the number of features in the strongly correlated feature subset.

5. The intelligent system integration method based on multimodal data fusion according to claim 1, characterized in that: The S7 specifically includes: S71. During the operation of the intelligent system, new data source input is received , the new data source input includes new modality data and its related features; S72. Input based on new data source , for existing integrated data Perform preliminary fusion processing to generate temporary fusion data : ； in, For independent modality datasets In the new data source The weight in For the new data source data of each modality, For independent modality datasets in existing integrated data The weight of To integrate data Middle data of each modality, is the number of modes in the new data source, is the number of modes in the existing integrated data; S73, based on the temporary fusion data , adaptively update the existing association rules, use incremental learning algorithms or other applicable methods to update the current association rule set Make real-time corrections to generate updated association rule sets : ； in, Based on temporary fusion data and new data sources Association rules The function to update. is the updated association rule; S74: According to the updated association rule set , recalculate the weight value of the modal data ; S75, according to the updated weight value , readjust the fusion strategy and generate a new fusion strategy .

Images