A method and system for intelligent fusion of heterogeneous multi-modal data

Abstract

The application provides a method and system for intelligent fusion of heterogeneous multi-modal data, comprising the following steps: monitoring whether the data source structure changes through a data monitoring model, automatically generating an adapter template if the data source structure changes, performing missing data optimization processing on the existing data source with serious missing if there is serious missing in the existing data source, and performing conflict data optimization processing on each pair of conflicting existing data sources if there is conflict in the existing data source. By using the method, the system can automatically generate an adapter template and dynamically expand a parsing strategy library, and can complete data cleaning, repairing and fusion without relying on a business system interface.

Description

Technical Field

[0001] This invention relates to the field of data processing, specifically to a method and system for intelligent fusion of heterogeneous multimodal data. Background Technology

[0002] Against the backdrop of rapid development in intelligent manufacturing and industrial digitalization, various sensors, monitoring devices, and data acquisition systems are widely deployed in various production systems to monitor key parameters in the production process in real time. With the continuous introduction of new sensing devices and data modalities, traditional methods struggle to dynamically adapt to changes, often relying on manual intervention or interface integration with business systems, resulting in poor system scalability and high maintenance costs. Furthermore, to avoid errors or temporary malfunctions in some sensors, multiple similar sensors are often deployed in the same monitoring area to collect the same data. However, existing data processing methods mostly employ simple summation or averaging techniques, lacking effective mechanisms for identifying and optimizing data source structure differences, conflicts, and missing data.

[0003] Therefore, there is an urgent need for a fusion method and system that can dynamically adapt to different modalities and automatically optimize the handling of conflicts and missing data from multiple sources. Summary of the Invention

[0004] To overcome existing technical problems, this invention provides a method and system for intelligent fusion of heterogeneous multimodal data that adapts to different data sources and optimizes conflicting and severely missing data sources.

[0005] The present invention adopts the following technical solution.

[0006] A method for intelligent fusion of heterogeneous multimodal data includes the following steps:

[0007] The data monitoring model monitors whether the data source structure has changed. If so, it determines whether the data is of the same modality or a new modality. If the data is of the same modality, an adapter template is automatically generated. If the data is of a new modality, the parsing strategy library is dynamically expanded and a new modality adapter template is generated.

[0008] Monitor existing data sources for conflicts or serious data gaps. If a serious data gap exists, perform data optimization on the data source with the serious data gap. If there are conflicts between existing data sources, perform conflict data optimization on each pair of conflicting existing data sources.

[0009] As a further improvement of the present invention, the data monitoring model monitors whether the data source structure has changed, and if so, determines whether the data is of the same modality or is a new modality. The specific steps include:

[0010] The data monitoring model stores multiple standard feature semantic libraries that correspond one-to-one with different data types, as well as multiple feature baseline sets that correspond one-to-one with existing data sources. The standard feature semantic library includes multiple standard parameter names; the feature baseline set includes at least one existing parameter name and existing parameter unit that correspond one-to-one with the standard parameter name.

[0011] When a data source is present, file type analysis is performed to distinguish the data types. If image data, video data, and audio data are present, metadata is extracted and treated as the input data source. If sensor data or text data are present, sensor data or text data is treated as the input data source. The corresponding standard feature semantic library is selected according to the data type. Features are extracted from the input data source using NLP (Natural Language Processing) technology to obtain the input feature name and input feature unit. The similarity between the input feature name and the standard parameter name under the selected standard feature semantic library is calculated. If the similarity is greater than the name similarity threshold, the input feature name is bound to the standard parameter name.

[0012] If there is no existing parameter name corresponding to the bound standard parameter name, then the input data source is determined to be new modal data;

[0013] If each bound standard parameter name has a corresponding existing parameter name, it is considered to be homomodal data;

[0014] It checks whether each input feature unit is the same as the existing parameter unit with the same standard parameter name, and checks whether each corresponding existing parameter name is under the same feature baseline set. If they are different units, it marks the unit conversion label. If they are not under the same feature baseline set, it marks the combination label. Otherwise, it is considered that the data source structure has not changed.

[0015] Record the device code of the input data source, and no longer monitor the data source structure changes for data sources with recorded device codes.

[0016] As a further improvement of the present invention, the data monitoring model is preset with multiple pairs of unit conversion data and a unit conversion factor corresponding to each pair of unit conversion data.

[0017] If the data of the same modality changes, the specific steps for automatically generating the adapter template include:

[0018] If a unit conversion tag is present, then the unit conversion function should be imported into the corresponding adapter. If there is a combined tag, multiple existing data sources are selected. Each bound standard parameter name has a corresponding existing parameter name in these multiple existing data sources. The adapters of the selected multiple existing data sources are integrated to form a new adapter.

[0019] As a further improvement to the present invention, the specific steps for dynamically expanding the parsing strategy library include:

[0020] For new modal data with text data type, extract at least one part-of-speech tag using NLP (Natural Language Processing) techniques. and semantic feature vector And extract the syntax tree depth and word frequency distribution parameters Integrate to form a text feature set ;

[0021] For new modal data such as image or video data, convolutional feature maps are extracted using ResNet-50. , obtained through global pooling And extract the HSV histogram statistics. and edge density , integration and formation ;

[0022] For new modal data of audio type, the MFCC coefficient matrix is ​​extracted using Librosa. Where n is the number of MFCC coefficients, T is the number of time frames, and the mean is calculated. Standard deviation and spectral centroid , integration and formation ;

[0023] Integration forms combined features ;

[0024] according to Generate parsing strategy units.

[0025] As a further improvement to the present invention, according to The specific steps for generating the parsing strategy unit include:

[0026] For text data, according to Different part-of-speech tags are processed by the corresponding NER entity recognition model to obtain part-of-speech tagging rules. ;

[0027] For image and video data, define a ROI detection threshold. To determine the key regions of the image

[0028] For audio data, set a threshold for determining silence segments. Identify and remove silent portions of the audio;

[0029] Integrate to form combinatorial analysis rules Using a hash function to Mapping to the corresponding combination parsing rules in the hash table The location.

[0030] As a further improvement of the present invention, the specific steps for generating a new modality adapter template include: selecting the adapter of the existing data source that corresponds most frequently with the existing parameter names and input feature names, and adjusting the core parameters of the adapter;

[0031] The specific steps for adjusting the adapter's core parameters include:

[0032] according to Constructing text modality processing functions ,in, This is the xth part-of-speech tag. ID corresponds to the NER entity recognition model ID for that part-of-speech tag. For each NER entity recognition model ID, the corresponding entity tag set is extracted. Calculate the dimension of the fully connected layer

[0033] according to Dynamic sampling to extract keyframe sets of video data keyframe set Processed as image data;

[0034] according to Constructing image data processing functions Calculate the convolution kernel size ,in, It is a scaling factor;

[0035] For audio data processing, applications The silence threshold, setting the step size of the recurrent neural network. · ,in, It is the sampling rate coefficient.

[0036] As a further improvement to the present invention, the specific steps for monitoring whether existing data sources are severely missing include:

[0037] Assume the data flow of the existing data source is ;

[0038] Calculate the timestamp interval, if it exists If so, it is assumed that the existing data source is missing. It is the data interval factor. This is the interval at which the existing data source is normally transmitted;

[0039] like If the missing values ​​are found, the existing data source is considered to have serious deficiencies. The existing data sources with serious deficiencies are selected and integrated to form a missing dataset. Otherwise, the missing values ​​of the existing data sources are estimated by interpolation.

[0040] As a further improvement of the present invention, while binding the input feature name to the standard parameter name, a standard name label is marked, and existing data sources with the same standard name label are integrated to form a pre-association set;

[0041] Calculate the Euclidean spatial distance between the k-th existing data source and the l-th existing data source in this pre-association set. ,when Greater than the preset maximum effective distance When the conflict calculation between the kth existing data source and the lth existing data source is ignored, otherwise the kth existing data source and the lth existing data source are extracted as a pair and integrated to form a preliminary conflict set;

[0042] The specific steps for detecting data source conflicts include:

[0043] Extract the existing data source with the highest transmission frequency from the initial collision set as the benchmark, and obtain the timestamp sequence. For the data sequence of each existing data source under the initial conflict set All continuous functions were generated using the cubic spline interpolation algorithm to obtain the conflict calculation set;

[0044] Determine the degree of conflict for each pair of existing data sources in the conflict computation set. calculate;

[0045] If the degree of conflict Greater than the dynamic conflict threshold If a conflict exists between the existing data sources, then each pair of existing data sources with a conflict is integrated to form a conflict dataset.

[0046] If the degree of conflict Greater than the maximum conflict threshold Among them, the maximum conflict threshold Greater than the dynamic conflict threshold If so, skip this pair of existing data sources and mark both existing data sources as warning.

[0047] As a further improvement of the present invention, the specific steps for optimizing the missing data of a severely missing existing data source include: reconstructing the average value of the missing data of the existing data source using the DTW reconstruction algorithm to obtain the reconstructed average value. ;

[0048] The specific steps for conflict data optimization for each pair of existing conflicting data sources include:

[0049] Extract the most recent historical data sequence from existing data sources and calculate the historical data deviation for each existing data source.

[0050] If the historical data deviation of both existing data sources is less than the corresponding deviation conflict threshold Then, mean correction and fusion are performed on these two data points;

[0051] If only one existing data source has historical data deviation greater than the corresponding deviation conflict threshold If the weight is greater than the corresponding deviation conflict threshold, then weight correction and fusion will be performed. The existing data source is marked as a warning;

[0052] If the historical data deviation of both existing data sources is greater than the corresponding deviation conflict threshold If so, skip the data optimization process for the existing data source and mark both existing data sources as warning.

[0053] Calculate the historical conflict degree between the q-th existing data source and the p-th existing data source. Adjust the deviation from the conflict threshold .

[0054] This invention also proposes a system for intelligent fusion of heterogeneous multimodal data, employing a method for intelligent fusion of heterogeneous multimodal data as described above, including: a data monitoring model, which includes:

[0055] The data collection module is used to collect data sources;

[0056] The data analysis module is used to analyze whether the data source structure has changed from the same modality or is a new modality.

[0057] The adapter generation module generates adapter templates based on the analysis results from the data analysis module.

[0058] The parsing strategy library generation module is used to dynamically expand the parsing strategy library for new modal data and send it to the adapter generation module;

[0059] The multi-source data analysis module is used to detect whether there are conflicts or serious omissions in existing data sources;

[0060] The data optimization module performs missing data optimization on severely missing existing data sources based on the analysis results of the multi-source data analysis module, or performs conflict data optimization on each pair of conflicting existing data sources.

[0061] The beneficial effects of this invention are as follows: By constructing a data monitoring model, real-time perception of structural changes in heterogeneous multimodal data sources is achieved, and corresponding adaptive processing mechanisms are adopted according to different change types. When a change in the same modal data structure is detected, the system can automatically generate an adapter template to ensure the continuity and accuracy of data access; when a new modal data source is identified, the system can dynamically expand the parsing strategy library and generate a new modal adapter template, enhancing the system's scalability. Simultaneously, the system possesses the ability to automatically identify and optimize multi-source data conflicts and severe data gaps, completing data cleaning, repair, and fusion without relying on business system interfaces, significantly improving data consistency, integrity, and availability. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0064] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product.

[0065] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings. The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0066] Reference Figure 1 A method for intelligent fusion of heterogeneous multimodal data includes the following steps:

[0067] The data monitoring model monitors whether the data source structure has changed. If so, it determines whether the data is of the same modality or a new modality. If the data is of the same modality, an adapter template is automatically generated. If the data is of a new modality, the parsing strategy library is dynamically expanded and a new modality adapter template is generated.

[0068] Specifically, the data monitoring model is built upon the MAML meta-learning model. The core idea of ​​MAML is to find initial parameters that allow the model to quickly adapt to new tasks after a small number of gradient updates. The logic for generating the subsequent adapter is similar. Traditional methods often rely on manual configuration or system reconfiguration when sensors or data modalities change, resulting in poor scalability and high maintenance workload. This invention achieves fully automatic dynamic adaptation by automatically judging changes in the data source structure and intelligently generating adapter templates or expanding the strategy library. This significantly reduces the need for manual intervention, enabling the system to seamlessly connect to new sensors or data sources.

[0069] As a further improvement of the present invention, the data monitoring model monitors whether the data source structure has changed, and if so, determines whether the data is of the same modality or is a new modality. The specific steps include:

[0070] The data monitoring model stores multiple standard feature semantic libraries that correspond one-to-one with different data types, as well as multiple feature baseline sets that correspond one-to-one with existing data sources. The standard feature semantic library includes multiple standard parameter names; the feature baseline set includes at least one existing parameter name and existing parameter unit that correspond one-to-one with the standard parameter name.

[0071] When a data source is present, file type analysis is performed to distinguish the data types. If image data, video data, and audio data are present, metadata is extracted and treated as the input data source. If sensor data or text data are present, sensor data or text data is treated as the input data source. The corresponding standard feature semantic library is selected according to the data type. Features are extracted from the input data source using NLP (Natural Language Processing) technology to obtain the input feature name and input feature unit. The similarity between the input feature name and the standard parameter name under the selected standard feature semantic library is calculated. If the similarity is greater than the name similarity threshold, the input feature name is bound to the standard parameter name.

[0072] Specifically, file type analysis includes sequentially performing file header analysis, file extension detection, and MIME type detection. If new data formats are subsequently introduced, additional file type analysis methods can be added accordingly. By distinguishing data types through file type analysis, preliminary modality segmentation of the data source input can be achieved, enabling subsequent comparison of standard feature names within the same data type. For example, when an image data source is input, file type analysis determines that the input is image data. Metadata is extracted, and features such as resolution are extracted. The resulting input feature names are then compared with the corresponding standard parameter names of the image data, and the similarity is calculated and bound. This avoids meaningless comparisons of massive amounts of standard parameter names between image data and sensor data, significantly reducing comparison time. Furthermore, when judging new modal data, the differences in data types can be comprehensively considered. Additionally, the existing parameter names in this invention are standardized into bound standard parameter names after the input data source is connected, and these input feature names are marked as existing parameter names. This allows for direct identification of the corresponding existing parameter names when new data sources are input. In addition to extracting features from the input data source using NLP (Natural Language Processing) technology, a feature type identification module can be combined. This module categorizes the features of the input data source into numerical, categorical, or temporal types, etc. By incorporating a feature type identification module, subsequent judgments regarding new modal data or changes in the same modality can be made based on feature type, similar to the data type classification mentioned above. This ensures that the differences in feature types are comprehensively considered when making judgments about new modal data.

[0073] At this point, it's important to note that metadata describes the data source, explaining its basic attributes. It doesn't directly constitute the video, image, or audio content itself, but it provides crucial support for understanding, managing, and using the content. Specific examples include: resolution, frame rate, bit rate, recording time, device model, container format, copyright information, sampling rate, number of channels, bit depth, encoding format, etc. These are the input feature names; if no unit exists, the input feature unit is recorded as 0. Different devices often have the same meaning but different names, making it difficult to directly correlate input feature names with standard parameter names. Therefore, similarity calculation is used for identification. Specifically, the name is converted into a feature vector, and the cosine similarity between the two is calculated to determine if they are synonyms. Refer to patent CN114722154B for details; similarity calculation for identifying synonyms will not be elaborated upon here.

[0074] If there is no existing parameter name corresponding to the bound standard parameter name, then the input data source is determined to be new modal data;

[0075] The standard feature semantic library records the standard parameter names of most data types, while the existing parameter names are the parameter names of the existing data sources that the system is currently connected to. If there is no existing parameter name corresponding to the bound standard parameter name, it means that the current system has not yet connected data of that data type. In addition to resolution, frame rate, and bit rate as mentioned above, the standard parameter names also include: temperature, vibration, pressure, displacement, illumination, light intensity, gas concentration, humidity, current, and conductivity, etc.

[0076] If each bound standard parameter name has a corresponding existing parameter name, it is considered to be homomodal data;

[0077] If both exist, it means that the current system has already accessed this data type, and therefore it can be considered as data of the same modality, and the adapter can be applied directly.

[0078] It checks whether each input feature unit is the same as the existing parameter unit with the same standard parameter name, and checks whether each corresponding existing parameter name is under the same feature baseline set. If they are different units, it marks the unit conversion label. If they are not under the same feature baseline set, it marks the combination label. Otherwise, it is considered that the data source structure has not changed.

[0079] At this point, it should be noted that the data source being accessed is not limited to a single sensor, a single shooting module, or a single recording module. It may also be directly connected to another pre-built multimodal data system. In this case, the multimodal data system may simultaneously have a temperature sensor, a humidity sensor, and a vibration sensor. During data transmission, the system accessed may directly package and send the data from these sensors to the system of this invention. However, since the system at this time only has individual adapters for the temperature sensor, humidity sensor, or vibration sensor, and does not have an adapter that can simultaneously access these sensors, multiple adapters need to be integrated in the future.

[0080] Record the device code of the input data source, and no longer monitor the data source structure changes for data sources with recorded device codes.

[0081] As a further improvement of the present invention, the data monitoring model is preset with multiple pairs of unit conversion data and a unit conversion factor corresponding to each pair of unit conversion data.

[0082] The multiple unit conversion data specifically refers to meters and centimeters, meters and decimeters, kilometers per hour and meters per second, etc. The unit conversion factor is 10 or 1 / 10 of the corresponding meters and decimeters, and 3.6 or 0.2778 of the corresponding kilometers per hour and meters per second.

[0083] If the data of the same modality changes, the specific steps for automatically generating the adapter template include:

[0084] If a unit conversion tag is present, then the unit conversion function should be imported into the corresponding adapter. If there is a combined tag, select multiple existing data sources. Each bound standard parameter name has a corresponding existing parameter name in these multiple existing data sources. Integrate the adapters of the selected multiple existing data sources to form a new adapter.

[0085] ,

[0086] ,

[0087] in, It is the input feature unit corresponding to the unit conversion data. It is the existing parameter unit corresponding to the unit conversion data. It is the feature data of the input data source. It is the transformed feature data obtained by converting the feature data of the input data source according to the feature units of the existing data source. All are unit conversion factors.

[0088] This formula is derived based on existing unit conversion relationships. Whether n equals 1 or 2 depends on the relationship between the input feature unit and the existing parameter units. For example, if the input feature unit corresponds to meters and the existing parameter unit corresponds to decimeters, then n=1. =10, where m corresponds to the unit conversion data between meters and decimeters. If the input feature unit corresponds to decimeters and the existing parameter unit corresponds to meters, then n=2. =1 / 10. Similarly, The determination and similar.

[0089] As a further improvement to the present invention, the specific steps for dynamically expanding the parsing strategy library include:

[0090] For new modal data with text data type, extract at least one part-of-speech tag using NLP (Natural Language Processing) techniques. and semantic feature vector And extract the syntax tree depth and word frequency distribution parameters Integrate to form a text feature set ;

[0091] Extracting semantic feature vectors Syntax tree depth is used to map words to a high-dimensional vector space, capturing the semantic similarity between words. By analyzing the depth of dependency trees constructed from the grammatical relationships between sentence components, we can better understand sentence structure and the relationships between its components. (Word frequency distribution parameters) Analyzing the frequency of each word helps identify the text's theme, keywords, and other information. By parsing this content, the text data can be preprocessed uniformly, facilitating subsequent integration and use. It's important to note that since sensor data output signals are generally numerical values ​​with units, or structured text, this invention treats sensor data as text data. If a small portion of sensor data is transmitted via other methods, some preprocessing is required. Because there are numerous types and models of sensors globally, this invention will not elaborate on them all.

[0092] For new modal data such as image or video data, convolutional feature maps are extracted using ResNet-50. , obtained through global pooling And extract the HSV histogram statistics. and edge density , integration and formation ;

[0093] Specifically, for video data, several frames can be randomly selected every second for processing to extract features covering key attributes such as semantics, color, and contour. More specifically, convolutional feature maps are extracted using ResNet-50. Convolutional layers capture local to global features, while global pooling preserves spatial information and compresses it into vectors, thereby extracting semantic information from the image. HSV histogram statistics transform RGB to HSV space, statistically analyze the histogram distribution of each channel, resist interference from lighting variations, and thus quantify the image's hue, saturation, and brightness. Edge density... Edges are detected using operators such as Canny, and the percentage of edge pixels per unit area is calculated.

[0094] For new modal data of audio type, the MFCC coefficient matrix is ​​extracted using Librosa. Where n is the number of MFCC coefficients, T is the number of time frames, and the mean is calculated. Standard deviation and spectral centroid , integration and formation ;

[0095] These three characteristics represent the energy distribution, fluctuation characteristics, and spectral center of audio, respectively, and can resist environmental noise interference and compress data volume.

[0096] Integration forms combined features This facilitates the subsequent generation of parsing strategy units and the generation of new modal adapter templates based on the parsing strategy units.

[0097] according to Generate parsing strategy units.

[0098] As a further improvement to the present invention, according to The specific steps for generating the parsing strategy unit include:

[0099] For text data, according to Different part-of-speech tags are processed by the corresponding NER entity recognition model to obtain part-of-speech tagging rules. By using part-of-speech tag dynamic matching NER model, the accuracy of entity recognition can be improved. Part-of-speech tags include nouns, verbs, etc.

[0100] For image and video data, define a ROI detection threshold. To determine the key regions of the image, ,in, It detects the maximum edge strength value using an edge detection algorithm. It is an edge detection factor; by using an edge density threshold, it identifies key regions, reducing redundant data processing by 90%. Specifically, This value is used to adjust the sensitivity of the ROI region. A higher value tends to capture the most prominent edges, which is suitable for simpler structural scenarios. When the value approaches 0.5, it tends to capture more edges, which is suitable for complex scenarios. For example, to monitor whether the current circuit board is bent or dented, the value is increased accordingly, such as to 0.85, to avoid capturing non-critical areas unrelated to the circuit board.

[0101] For audio data, set a threshold for determining silence segments. Identify and remove silent portions of the audio. Specifically, ,in, These are constraint values; thus, effective audio is extracted, enhancing the ability for subsequent analysis. In order to prevent A value less than 0 will cause an error or delete all audio. Specifically, The mean is 0.1. Subtract twice the standard deviation It follows the two-standard-deviation law, with 95% of the values ​​falling within this range. In factory applications, it can eliminate environmental noise, such as vibrations from machine operation or wind noise.

[0102] Integrate to form combinatorial analysis rules Using a hash function to Mapping to the corresponding combination parsing rules in the hash table The location.

[0103] Convenient to combine features Quickly locate the combination parsing rules .

[0104] As a further improvement of the present invention, the specific steps for generating a new modality adapter template include: selecting the adapter of the existing data source that corresponds most frequently with the existing parameter names and input feature names, and adjusting the core parameters of the adapter;

[0105] Since the adapter consists of multiple aspects and components, the core of this invention lies in multi-data fusion processing. Most existing heterogeneous multimodal data processing is done through the adapter, so this invention will not elaborate on the other parts of the adapter here.

[0106] The specific steps for adjusting the adapter's core parameters include:

[0107] according to Constructing text modality processing functions ,in, This is the xth part-of-speech tag. ID corresponds to the NER entity recognition model ID for that part-of-speech tag. For each NER entity recognition model ID, the corresponding entity tag set is extracted. Calculate the dimension of the fully connected layer ,in, These are empirical coefficients, where x represents the x-th NER model;

[0108] Used to quantify the semantic complexity of the current text data, when the entity tag set The more dimensions a fully connected layer has, the stronger its representational power needs to be, thus requiring an increase in the layer's dimensionality. Specifically, It can be determined directly through string calculations, using empirical coefficients. The default value is 1. If the model is too simple, the empirical coefficient can be increased appropriately. The value of , and conversely, the empirical coefficient can be reduced. The value of .

[0109] according to Dynamic sampling to extract keyframe sets of video data ,in, It is the edge intensity of the t-th frame, which is the keyframe set. Processed as image data;

[0110] Since any change in motion in a video, such as object movement or camera module movement, will cause edge changes, filtering out frames with insufficient edge strength can not only compress the amount of data, but also facilitate subsequent analysis.

[0111] according to Constructing image data processing functions ,in, = , yes Calculate the convolution kernel size ,in, It is a scaling factor;

[0112] Large convolutional kernels are primarily used to capture macroscopic defects, requiring fewer scans, while small convolutional kernels are used to focus on microscopic anomalies, but require more scans. Furthermore, different imaging modules have varying edge intensity values. Therefore, calculating the convolutional kernel size using edge intensity values ​​can balance defect detection accuracy and computational efficiency. Specifically, the scale factor... The resolution can be obtained by dividing the size of the smallest defect of the monitored object by the resolution. For example, if the smallest defect of the monitored object is a 3cm crack, the imaging resolution of the imaging module is 0.3cm / px. The value is 10. Of course, a weighting coefficient can also be introduced in this calculation. This is due to the scaling factor. The determination of the minimum defect of the object to be tested needs to be determined manually. Therefore, it can be set to 10 by default, so that the input data source can automatically generate an adapter. It can then be manually adjusted later to still achieve the technical effect of automatically generating an adapter.

[0113] For audio data processing, applications The silence threshold, setting the step size of the recurrent neural network. · ,in, This is the sampling rate coefficient. When the mute threshold... A higher value indicates greater background noise, with effective audio primarily concentrated in high-energy segments, while the silence threshold... A lower value indicates a quieter environment and a more dispersed effective audio distribution. Setting a fixed step size cannot adapt to dynamic changes in energy distribution; for example, in high-noise environments, it's often necessary to capture short-lived fault sounds, such as equipment explosions. Therefore, this formula allows for the determination of the silence threshold. When the value is high, shorten the step size. This allows the RNN to perform dense scanning, avoiding missing transient high-frequency faults. Conversely, increasing the step size... This saves resources.

[0114] Specifically, It can be set to 0.002, at the mute threshold. When it is 0.9, =2.2ms; at the mute threshold When it is 0.3, =6.6ms.

[0115] Monitor existing data sources for conflicts or serious data gaps. If a serious data gap exists, perform data optimization on the data source with the serious data gap. If there are conflicts between existing data sources, perform conflict data optimization on each pair of conflicting existing data sources.

[0116] At this point, it's important to note that the existing data sources for conflict mentioned here refer to existing data of the same detection object. For example, multiple vibration sensors might be installed in the same area, or multiple temperature and humidity sensors might be present in a room. Because image, video, and audio data have strong spatiotemporal characteristics, conflict calculations are generally not performed on them. Subsequent steps will primarily focus on calculating conflicts using sensor data.

[0117] As a further improvement to the present invention, the specific steps for monitoring whether existing data sources are severely missing include:

[0118] Assume the data flow of the existing data source is ,in, It's a timestamp. It is d-dimensional data;

[0119] Calculate the timestamp interval, if it exists If so, it is assumed that the existing data source is missing. It is the data interval factor. This is the interval at which the existing data source is normally transmitted;

[0120] like If the missing values ​​are found, the existing data source is considered to have serious deficiencies. The existing data sources with serious deficiencies are selected and integrated to form a missing dataset. Otherwise, the missing values ​​of the existing data sources are estimated by interpolation.

[0121] Specifically, data interval factor The specific value can be referenced to the interval time of a normal signal interruption, such as five seconds. Greater than 5 seconds. And the data interval factor... The specific value should be determined by adding the device's power-off restart time to the normal signal interruption time. For example, if a device takes ten seconds to restart and the normal signal interruption time is five seconds, then the data interval... More than 15 seconds.

[0122] As a further improvement of the present invention, while binding the input feature name to the standard parameter name, a standard name label is marked, and existing data sources with the same standard name label are integrated to form a pre-association set;

[0123] Calculate the Euclidean spatial distance between the k-th existing data source and the l-th existing data source in this pre-association set. ,when Greater than the preset maximum effective distance When the conflict calculation between the kth existing data source and the lth existing data source is ignored, otherwise the kth existing data source and the lth existing data source are extracted as a pair and integrated to form a preliminary conflict set;

[0124] When multiple vibration sensors are installed in the same area, and the area is large, the data detected by existing data sources located at two opposite corners of the area may have a large spatial difference. In this case, there is no need to perform conflict calculation, because the vibration source may be close to one of them, resulting in one detecting a higher vibration value and the other a lower value.

[0125] Furthermore, conditions can be added based on actual needs, such as introducing height distance calculations, even for Euclidean spatial distances. Less than the preset maximum effective distance However, when there is a significant difference in the height of the two existing data sources, the conflict calculation between the two existing data sources can be ignored.

[0126] The specific steps for detecting data source conflicts include:

[0127] Extract the existing data source with the highest transmission frequency from the initial collision set as the benchmark, and obtain the timestamp sequence. For the data sequence of each existing data source under the initial conflict set All functions are generated using cubic spline interpolation to obtain the conflict calculation set:

[0128] ,

[0129] in, This represents the data of the k-th existing data source in the initial conflict set after interpolation at a unified timestamp t;

[0130] This method enables the "unification" of timestamps from each existing data source, thereby facilitating subsequent data computation.

[0131] Determine the degree of conflict for each pair of existing data sources in the conflict computation set. calculate:

[0132] ,

[0133] in, represents the degree of conflict between the p-th and q-th existing data sources, where n is the length of the timestamp sequence. It is the weight coefficient of the i-th time point in the same timestamp sequence. It is the time-based weight value. It is the latest time point in the same timestamp sequence. It is the i-th time point in the same timestamp sequence;

[0134] If the degree of conflict Greater than the dynamic conflict threshold If a conflict exists between the existing data sources, then each pair of existing data sources with a conflict is integrated to form a conflict dataset.

[0135] If the degree of conflict Greater than the maximum conflict threshold Among them, the maximum conflict threshold Greater than the dynamic conflict threshold If so, skip this pair of existing data sources and mark both existing data sources as warning.

[0136] Because each sensor's data inevitably contains manufacturing errors, assembly errors, and installation errors, the detection values ​​of different sensors will vary. Therefore, a dynamic conflict threshold is set. This avoids performing conflict data optimization for every pair of existing data sources. Specifically, a dynamic conflict threshold... The dynamic collision threshold is determined based on the detection accuracy of each sensor model. For example, if the accuracy of an infrared distance sensor is 0.01mm, then the dynamic collision threshold is... It can be set to 0.02. For easier data processing, the value of the time-based weight can be adjusted accordingly to adjust the dynamic conflict threshold. Set to 2. Additionally, the time-based weight value is determined based on the timestamp sequence and the dynamic conflict threshold. The magnification factor is determined by the time stamp sequence. For example, if the oldest time point in the timestamp sequence is 23,000 milliseconds and the newest time point is 23,200 milliseconds, and the length of the timestamp sequence is 200, then the weighting factor... The average denominator is 100, and the dynamic conflict threshold is... If magnified 100 times, the time base weight value is 10000.

[0137] Set the maximum conflict threshold It is used to distinguish different usage environments, and European-style spatial distance Less than the maximum effective distance Sensor data, for example, if two existing data sources are temperature sensors, one in a refrigerated indoor space and the other outdoors in direct sunlight, and both are mounted on opposite sides of the same wall, then due to their Euclidean spatial distance... Less than the preset maximum effective distance And it has a high degree of conflict. If these conditions are not met, the data source will be marked as a warning, allowing monitoring personnel to manually add labels to both data sources, such as "outdoor" and "indoor," treating them as standard name labels. If multiple temperature sensors are subsequently installed outdoors, manually adding the "outdoor" label will allow existing data sources with temperature detection labels and outdoor labels to be integrated into a pre-association set.

[0138] Maximum conflict threshold The calculation is based on the normal value fluctuations in the region. If the value exceeds the normal value fluctuations, it can be considered that the two existing data sources are in different environments, and thus marked as a warning state for subsequent manual labeling.

[0139] As a further improvement of the present invention, the specific steps for optimizing the missing data of a severely missing existing data source include: reconstructing the average value of the missing data of the existing data source using the DTW reconstruction algorithm to obtain the reconstructed average value. ;

[0140] Specifically, the reconstructed mean is obtained through the DTW reconstruction algorithm. The expression is as follows:

[0141] ,

[0142] ,

[0143] in, It is at the midpoint of the missing time period The reconstructed mean, It is the number of neighboring data points. It is the dynamic data weight coefficient of the i-th neighboring data point. It is the data vector of the i-th neighboring data point. It is the number of similar historical data segments. It is the dynamic data weight coefficient of the j-th historical similar data segment. It is the data vector of the j-th historical similar data segment. It is the midpoint of the missing time period. The time distance to the i-th nearest data point. It is the maximum allowed nearest neighbor distance. It is the similarity attenuation distance coefficient. It is the midpoint between the j-th historical similar data segment and the missing time period. DTW distance.

[0144] When data sources are severely missing, reconstructing each missing data point would consume a massive amount of computation. Therefore, directly calculating an average can significantly reduce the computational load. The number of neighboring data points is determined based on the number of missing data points, aiming to make the number of missing data points approximate the number of neighboring data points. For the algorithm used to calculate historical similar data segments and DTW distance, refer to patent CN113225209B. It's important to note that since there is no specific data for the missing time period, DTW distance is also calculated using data from neighboring data points, but the time is still based on the midpoint of the missing time period. To calculate.

[0145] By calculating neighboring data points and similar historical data segments, it's possible not only to capture the data change trends of neighboring data points and thus estimate the approximate value of missing data, but also to capture periodic numerical changes. For example, temperature sensor data is severely lacking during the evening period, and... It can capture the trend of temperature decreasing at night and increasing during the day from the temperature sensor, and then comprehensively fit the severely missing values. For the dynamic data weighting coefficients of adjacent data points, since the time corresponding to adjacent data points is adjacent to the missing time period, it exhibits a relatively obvious linear change; therefore, the dynamic data weighting coefficients also adopt a linear change. For the dynamic data weighting coefficients of historical similar data segments, since the DTW distance is used to calculate the similarity with historical similar data segments, the reference value decreases sharply as the similarity decreases. Therefore, an e-index is set for dynamic adjustment. When the DTW distance is larger, i.e., the similarity is lower, the dynamic data weighting coefficient value of historical similar data segments decreases sharply.

[0146] In one specific embodiment of the present invention, a temperature sensor experienced data loss on the evening of April 1, 2025. The missing data period was from 18:00 to 19:00, assuming one data point every 5 minutes, the number of missing data points was 12. (Midpoint) It's 18:30 now, and needs to be restructured. Mean temperature at any given time. Number of neighboring data points. The maximum allowable nearest neighbor distance is 3. It is 30 minutes, the similarity decay distance coefficient is 4, and the number of historical similar data segments. 2. Select the nearest points before and after the missing segment, and calculate... ;

[0147] .

[0148] The specific steps for conflict data optimization for each pair of existing conflicting data sources include:

[0149] Extract the most recent historical data sequence from existing data sources and calculate the historical data deviation for each existing data source:

[0150] ,

[0151] ,

[0152] ,

[0153] in, Is the p-th existing data source in time? Interpolated data and historical time Data deviation, Is the qth existing data source in time? Interpolated data and historical time Data deviation, Timestamp sequences belonging to existing data sources used as a benchmark , It is the minimum historical data time interval of the p-th data source. It is the maximum historical data time interval of the p-th existing data source. It is the sampling frequency of the p-th existing data source. It is the minimum historical data time interval of the qth existing data source. It is the maximum historical data time interval of the qth existing data source. It is the sampling frequency of the qth existing data source. It represents the historical deviation of the p-th existing data source. It represents the historical deviation of the qth existing data source. It is a timestamp sequence Number of timestamps;

[0154] For each existing data source, based on the timestamp sequence The number of timestamps and the different sampling frequencies for each existing data source determine the historical time. The range of values ​​ensures that the interpolated data at a unified timestamp t can be compared one-to-one with historical data. Furthermore, calculating the deviation from historical data facilitates subsequent assessments of whether the existing data source exhibits anomalies.

[0155] If the historical data deviation of both existing data sources is less than the corresponding deviation conflict threshold Then, mean correction and fusion are performed on these two data points;

[0156] If only one existing data source has historical data deviation greater than the corresponding deviation conflict threshold If the weight is greater than the corresponding deviation conflict threshold, then weight correction and fusion will be performed. The existing data source is marked as a warning;

[0157] Specifically, the expression for weighted correction fusion is as follows:

[0158] ，

[0159] ,

[0160] in, and All are dynamic weighting coefficients. and These are the conflict weight coefficients;

[0161] More specifically, The value can be 2. The value is 1, when , and The numerical values ​​are adjusted, thereby reducing the deviation of historical data from exceeding the corresponding deviation and conflict threshold. The trust level of the existing data source makes the merged values ​​biased towards the existing data source with a normal deviation from historical data.

[0162] If the historical data deviation of both existing data sources is greater than the corresponding deviation conflict threshold If so, skip the data optimization process for the existing data source and mark both existing data sources as warning.

[0163] Calculate the historical conflict degree between the q-th existing data source and the p-th existing data source. Adjust the deviation from the conflict threshold ,

[0164] Specifically, adjust the deviation conflict threshold. The method is as follows:

[0165] ,

[0166] in, It is the basic conflict threshold between the qth existing data source and the pth existing data source.

[0167] Specifically, if there is no historical conflict Then the item It can be set to 0 by default, where The default value is 1. = , The determination is based on the detection frequency, precision measurement unit, and fluctuation range of different types of sensors, such as the temperature fluctuation of a temperature sensor not exceeding 5 degrees within ten seconds.

[0168] If an existing data source suddenly experiences a minor error that causes a data monitoring error, and its conflict level is below the maximum conflict threshold... And less than the dynamic conflict threshold At this point, conflict data optimization processing will begin, shifting the data towards more normal data. If one of the existing data sources exceeds the corresponding conflict deviation threshold... This indicates that the numerical fluctuations of the existing data source are abnormal, and weight correction, fusion, and maintenance should be performed to reduce the computational weight of the existing data source and make it biased towards normal data. If both existing data sources exceed the deviation conflict threshold... If the problem is not found, a manual inspection and repair should be carried out. If no problems are found, the fluctuation range can be considered to be within the normal range, and the basic conflict threshold for the existing data source should be increased accordingly.

[0169] In addition, it should be noted that if the qth existing data source undergoes conflict data optimization with multiple existing data sources, the results should be calculated separately, and then the average value should be taken.

[0170] For dynamically adjusting the deviation conflict threshold The calculation principle is as follows: if the existing data source has a high degree of conflict over a long period, it means that increasing the deviation conflict threshold can reduce false alarms for unstable data; if the conflict level has a low degree over a long period, it means that the two data sources are in harmony over a long period, and decreasing the deviation conflict threshold can improve the fusion accuracy. Specifically, the basic conflict threshold... Calculations are performed based on the normal fluctuation range of the existing data source, while also considering the degree of conflict. It is also calculated based on the differences between two existing data sources, therefore its numerical value is related to the basic conflict threshold. approximate, The default value is 1.

[0171] Thus, this invention enables fully automated and intelligent fusion processing of new data source inputs or new system access. From automatically and dynamically expanding the parsing strategy library and generating new modal adapter templates, to automatically detecting and handling data loss and conflicts, manual annotation is performed when some existing data sources have specific characteristics, significantly reducing workload. For example, when a system has already connected a certain number of data sources, and multiple temperature sensors are installed in a cold storage warehouse (multiple sensors both inside and outside), the existing temperature sensor adapter template can be directly applied when connecting any first temperature sensor. This is because the Euclidean space distance between the first installed temperature sensor and other temperature sensors... Much greater than the preset maximum effective distance Therefore, they will not be included in the pre-association set. When other temperature sensors are subsequently connected, they will gradually enter this pre-association set. When only the temperature sensors inside the warehouse are installed, these temperature sensors can automatically achieve data fusion, thereby obtaining a highly accurate environmental temperature assessment. When temperature sensors installed outside the warehouse are connected, the temperature difference will be marked as a warning state to remind operators to manually add tags. This makes the outdoor temperature sensors form their own pre-association set, while the indoor temperature sensors do not need to be tagged separately because they do not have the same standard name tag. It should be noted that the same standard name tag means that they completely conform to the standard name tag; having one less or one more tag does not belong to the same standard name tag. It should be noted that the warning state is used to remind operators to add tags. Even without reminders, operators should still add tags because the installed sensors are in different environments. For example, during a certain period in winter, the room temperature in the cold storage warehouse is similar to that outdoors. Installing indoor and outdoor temperature sensors during this period will lead to data errors in subsequent data fusion.

[0172] As a further improvement of the present invention, a heterogeneous multimodal data intelligent fusion method as described above is employed, comprising: a data monitoring model, wherein the data monitoring model includes:

[0173] The data collection module is used to collect data sources;

[0174] The data analysis module is used to analyze whether the data source structure has changed from the same modality or is a new modality.

[0175] The adapter generation module generates adapter templates based on the analysis results from the data analysis module.

[0176] The parsing strategy library generation module is used to dynamically expand the parsing strategy library for new modal data and send it to the adapter generation module;

[0177] The multi-source data analysis module is used to detect whether there are conflicts or serious omissions in existing data sources;

[0178] The data optimization module performs missing data optimization on severely missing existing data sources based on the analysis results of the multi-source data analysis module, or performs conflict data optimization on each pair of conflicting existing data sources.

[0179] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for intelligent fusion of heterogeneous multimodal data, characterized in that, Includes the following steps: The data monitoring model monitors whether the data source structure has changed. If so, it determines whether the change is due to changes in the same modality or the introduction of a new modality. If the change is due to changes in the same modality, the corresponding unit transformation label or combined label is obtained. If a unit transformation label exists, the corresponding adapter is selected to import the unit transformation function. If there is a combined tag, then select adapters from multiple existing data sources and integrate them to form a new adapter; If it is new modal data, the parsing strategy library is dynamically expanded and a new modal adapter template is generated. This process includes the following steps: dynamically parsing different data types to obtain combined parsing rules. Select a new modal adapter and apply it according to the combination analysis rules. Adjust the adapter's core parameters; Monitor existing data sources for conflicts or serious missing data. If there are serious missing data sources, perform missing data optimization on the existing data sources with serious missing data. If there are conflicts among existing data sources, perform conflict data optimization on each pair of conflicting existing data sources. The specific steps for monitoring whether existing data sources are severely missing include: Assume the data flow of the existing data source is ; Calculate the timestamp interval, if it exists If so, it is assumed that the existing data source is missing. It is the data interval factor. This is the interval at which the existing data source is normally transmitted; like If the missing values ​​are found, the existing data source is considered to have serious deficiencies. The existing data sources with serious deficiencies are selected and integrated to form a missing dataset. Otherwise, the missing values ​​of the existing data sources are estimated by interpolation. The specific steps for monitoring for conflicts in existing data sources include: Based on the Euclidean spatial distance between any two existing data sources An initial collision set is formed by integrating the data. The existing data source with the highest transmission frequency in the initial collision set is extracted as the benchmark, and the timestamp sequence is obtained. For the data sequence of each existing data source under the initial conflict set All continuous functions were generated using the cubic spline interpolation algorithm to obtain the conflict calculation set; Determine the degree of conflict for each pair of existing data sources in the conflict computation set. calculate; If the degree of conflict Greater than the dynamic conflict threshold If a conflict exists between the existing data sources, then each pair of existing data sources with a conflict is integrated to form a conflict dataset. If the degree of conflict Greater than the maximum conflict threshold Among them, the maximum conflict threshold Greater than the dynamic conflict threshold If so, skip this pair of existing data sources and mark both existing data sources as warning.

2. The method for intelligent fusion of heterogeneous multimodal data according to claim 1, characterized in that, The data monitoring model monitors whether the data source structure has changed. If so, the specific steps to determine whether the data is from the same modality or a new modality include: The data monitoring model stores multiple standard feature semantic libraries that correspond one-to-one with different data types, as well as multiple feature baseline sets that correspond one-to-one with existing data sources. The standard feature semantic library includes multiple standard parameter names; the feature baseline set includes at least one existing parameter name and existing parameter unit that correspond one-to-one with the standard parameter name. When a data source is present, file type analysis is performed to distinguish the data types. If image data, video data, and audio data are present, metadata is extracted and treated as the input data source. If sensor data or text data are present, sensor data or text data is treated as the input data source. The corresponding standard feature semantic library is selected according to the data type. Features are extracted from the input data source using NLP (Natural Language Processing) technology to obtain the input feature name and input feature unit. The similarity between the input feature name and the standard parameter name under the selected standard feature semantic library is calculated. If the similarity is greater than the name similarity threshold, the input feature name is bound to the standard parameter name. If there is no existing parameter name corresponding to the bound standard parameter name, then the input data source is determined to be new modal data; If each bound standard parameter name has a corresponding existing parameter name, it is considered to be homomodal data; It checks whether each input feature unit is the same as the existing parameter unit with the same standard parameter name, and checks whether each corresponding existing parameter name is under the same feature baseline set. If they are different units, it marks the unit conversion label. If they are not under the same feature baseline set, it marks the combination label. Otherwise, it is considered that the data source structure has not changed. Record the device code of the input data source, and no longer monitor the data source structure changes for data sources with recorded device codes.

3. The method for intelligent fusion of heterogeneous multimodal data according to claim 2, characterized in that, The data monitoring model is pre-set with multiple pairs of unit-transformed data and the corresponding unit transformation factor for each pair of unit-transformed data; The specific steps for integrating adapters from multiple existing data sources to form a new adapter include: Select multiple existing data sources. Each bound standard parameter name has a corresponding existing parameter name in these multiple existing data sources. Integrate the adapters of the selected multiple existing data sources to form a new adapter.

4. The method for intelligent fusion of heterogeneous multimodal data according to claim 2, characterized in that, The specific steps of dynamic parsing processing include: For new modal data with text data type, extract at least one part-of-speech tag using NLP (Natural Language Processing) techniques. and semantic feature vector And extract the syntax tree depth and word frequency distribution parameters Integrate to form a text feature set ; For new modal data such as image or video data, convolutional feature maps are extracted using ResNet-50. , obtained through global pooling And extract the HSV histogram statistics. and edge density , integration and formation ; For new modal data of audio type, the MFCC coefficient matrix is ​​extracted using Librosa. Where n is the number of MFCC coefficients, T is the number of time frames, and the mean is calculated. Standard deviation and spectral centroid , integration and formation ; Integration forms combined features ; according to Generate parsing strategy units.

5. The method for intelligent fusion of heterogeneous multimodal data according to claim 4, characterized in that, according to The specific steps for generating the parsing strategy unit include: For text data, according to Different part-of-speech tags are processed by the corresponding NER entity recognition model to obtain part-of-speech tagging rules. ; For image and video data, define a ROI detection threshold. To determine the key regions of the image; For audio data, set a threshold for determining silence segments. Identify and remove silent portions of the audio; Integrate to form combinatorial analysis rules Using a hash function to Mapping to the corresponding combination parsing rules in the hash table The location.

6. The method for intelligent fusion of heterogeneous multimodal data according to claim 5, characterized in that, The specific steps for selecting a new modality adapter include: selecting the adapter from the existing data source that corresponds most frequently to the existing parameter names and input feature names; According to the combinatorial analysis rules The specific steps for adjusting the adapter's core parameters include: according to Constructing text modality processing functions ,in, This is the xth part-of-speech tag. ID corresponds to the NER entity recognition model ID for that part-of-speech tag. For each NER entity recognition model ID, the corresponding entity tag set is extracted. Calculate the dimension of the fully connected layer ; according to Dynamic sampling to extract keyframe sets of video data keyframe set Processed as image data; according to Constructing image data processing functions Calculate the convolution kernel size in, It is a scaling factor; For audio data processing, applications The silence threshold, setting the step size of the recurrent neural network. · ,in, It is the sampling rate coefficient.

7. The method for intelligent fusion of heterogeneous multimodal data according to claim 2, characterized in that, While binding the input feature name to the standard parameter name, the standard name label is marked, and existing data sources with the same standard name label are integrated to form a pre-association set; Based on the Euclidean spatial distance between any two existing data sources The specific steps for integrating and forming a preliminary conflict set include: Calculate the Euclidean spatial distance between the k-th existing data source and the l-th existing data source in this pre-association set. ,when Greater than the preset maximum effective distance If the conflict calculation between the kth existing data source and the lth existing data source is ignored, otherwise the kth existing data source and the lth existing data source are extracted as a pair and integrated to form a preliminary conflict set.

8. The method for intelligent fusion of heterogeneous multimodal data according to claim 7, characterized in that, The specific steps for optimizing existing data sources with severe missing data include: reconstructing the average value of the missing data from the existing data source using the DTW reconstruction algorithm, and obtaining the reconstructed mean. ; The specific steps for conflict data optimization for each pair of existing conflicting data sources include: Extract the most recent historical data sequence from existing data sources and calculate the historical data deviation for each existing data source. If the historical data deviation of both existing data sources is less than the corresponding deviation conflict threshold Then, mean correction and fusion are performed on these two data points; If only one existing data source has historical data deviation greater than the corresponding deviation conflict threshold If the weight is greater than the corresponding deviation conflict threshold, then weight correction and fusion will be performed. The existing data source is marked as a warning; If the historical data deviation of both existing data sources is greater than the corresponding deviation conflict threshold If so, skip the data optimization process for the existing data source and mark both existing data sources as warning. Calculate the historical conflict degree between the q-th existing data source and the p-th existing data source. Adjust the deviation from the conflict threshold .

9. A system for intelligent fusion of heterogeneous multimodal data, characterized in that, The method for intelligent fusion of heterogeneous multimodal data according to any one of claims 1-8 includes: a data monitoring model, wherein the data monitoring model includes: The data collection module is used to collect data sources; The data analysis module is used to analyze whether the data source structure has changed from the same modality or is a new modality. The adapter generation module generates adapter templates based on the analysis results from the data analysis module. The parsing strategy library generation module is used to dynamically expand the parsing strategy library for new modal data and send it to the adapter generation module; The multi-source data analysis module is used to detect whether there are conflicts or serious omissions in existing data sources; The data optimization module performs missing data optimization on severely missing existing data sources based on the analysis results of the multi-source data analysis module, or performs conflict data optimization on each pair of conflicting existing data sources.

Images