Method and system for intelligently fusing heterogeneous multi-modal data

Abstract

The invention provides a heterogeneous multi-modal data intelligent fusion method and system, and the method comprises the following steps: monitoring whether a data source structure is changed or not through a data monitoring model, and automatically generating an adapter template if the data source structure is changed, if the existing data sources are seriously missing, missing data optimization processing is carried out on the existing data sources which are seriously missing, and if existing data source conflicts exist, conflicting data optimization processing is carried out on each pair of conflicting existing data sources. By the adoption of the method, the system can automatically generate the adapter template and the dynamic expansion analysis strategy library, and data cleaning, repairing and fusion can be completed without depending on a service system interface.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and system for intelligent fusion of heterogeneous multimodal data. Background Art

[0002] Against the backdrop of the rapid development of intelligent manufacturing and industrial digitalization, a wide range of sensors, monitoring equipment, and data acquisition systems have been widely deployed in various production systems to monitor key parameters in real time during the production process. With the continuous introduction of new sensor devices and data modalities, traditional methods have difficulty dynamically adapting to changes and often rely on manual intervention or business system interface docking, resulting in poor system scalability and high maintenance costs. Furthermore, to avoid errors or temporary failures in some sensors, multiple similar sensors are often deployed in the same monitoring area to collect the same data. However, existing data processing methods often use crude fusion methods such as simple summation or averaging, lacking effective identification and optimization mechanisms for data source structure differences, conflicts, and missing data.

[0003] Therefore, there is an urgent need for a fusion method and system that can dynamically adapt to different modalities and automatically optimize the processing of conflicts and missing multi-source data. Summary of the Invention

[0004] To overcome existing technical problems, the present invention provides a method and system for intelligent fusion of heterogeneous multimodal data that adapts to different data sources and optimizes data from conflicting and severely missing data sources.

[0005] The present invention adopts the following technical solutions.

[0006] A method for intelligent fusion of heterogeneous multimodal data, comprising the following steps: The data monitoring model is used to monitor whether the data source structure has changed. If so, it is determined whether the change is to the same modality data or to new modality data. If the change is to the same modality data, an adapter template is automatically generated. If the change is to new modality data, the parsing strategy library is dynamically expanded and a new modality adapter template is generated. Monitor existing data sources for conflicts or serious omissions. If there are serious omissions in existing data sources, optimize the missing data for the seriously missing existing data sources. If there are conflicts in existing data sources, optimize the conflicting data for each pair of conflicting existing data sources.

[0007] As a further improvement of the present invention, the data monitoring model monitors whether the data source structure has changed. If so, the specific steps of determining whether the change is to the same modality data or to new modality data include: The data monitoring model stores multiple standard feature semantic libraries corresponding to different data types, and multiple feature baseline sets corresponding to existing data sources. The standard feature semantic libraries include multiple standard parameter names; the feature baseline sets include at least one existing parameter name and existing parameter unit corresponding to the standard parameter name. When there is a data source input, file type analysis is performed to distinguish the data type. If image data, video data, and audio data exist, metadata is extracted and regarded as the input data source. If sensor data or text data exists, the sensor data or text data is regarded as the input data source. According to the data type, the corresponding standard feature semantic library is selected, and features of the input data source are extracted using NLP natural language technology to obtain input feature names and input feature units. The input feature names are calculated one by one with the standard parameter names under the selected standard feature semantic library. If the similarity is greater than the name similarity threshold, the input feature name is bound to the standard parameter name. If there is no existing parameter name corresponding to the bound standard parameter name, it is determined that the input data source is new modal data; If each bound standard parameter name has a corresponding existing parameter name, it is considered to be homomodal data; Check whether each input feature unit is the same as the existing parameter unit corresponding to the same standard parameter name, and check whether each corresponding existing parameter name is in the same feature baseline set. If they are different units, mark the unit conversion label. If they are not in the same feature baseline set, mark the combination label. Otherwise, it is assumed that the data source structure has not changed. Record the device code of the input data source, and no longer monitor the data source structure changes for the data source with the recorded device code.

[0008] As a further improvement of the present invention, the data monitoring model is preset with multiple pairs of unit conversion data and unit conversion factors corresponding to each pair of unit conversion data; The specific steps for automatically generating an adapter template if the same modality data changes include: If there is a unit conversion tag, introduce the unit conversion function in the corresponding adapter If there is a combination tag, multiple existing data sources are selected, and each bound standard parameter name has a corresponding existing parameter name in these multiple existing data sources. The adapters of the selected multiple existing data sources are integrated to form a new adapter.

[0009] As a further improvement of the present invention, the specific steps of dynamically expanding the parsing strategy library include: For new modal data whose data type is text data, extract at least one part-of-speech tag through NLP natural language processing technology and semantic feature vector , and extract the syntax tree depth and word frequency distribution parameters , integrated to form a text feature set ; For new modal data whose data type is image data or video data, convolutional feature maps are extracted through ResNet-50 , obtained by global pooling , and extract HSV histogram statistics and edge density , integrated to form ; For new modal data whose data type is audio data, extract the MFCC coefficient matrix through Librosa , where n is the number of MFCC coefficients and T is the number of time frames, the mean is calculated , standard deviation and spectrum centroid , integrated to form ; Integration to form combined features ; according to Generates a parsing strategy unit.

[0010] As a further improvement of the present invention, according to The specific steps to generate a parsing strategy unit include: For the data in this article, according to Different part-of-speech tags in the NER entity recognition model are processed to obtain the part-of-speech tagging rules. ; For image and video data, define the ROI detection threshold To identify key areas of the image; For audio data, set the silence segment judgment threshold Determine the silent part in the audio and delete it; Integrate to form combined parsing rules , through the hash function Mapped to the corresponding combination parsing rules in the hash table location.

[0011] As a further improvement of the present invention, the specific steps of generating a new modality adapter template include: selecting an adapter of an existing data source having the most correspondence between existing parameter names and input feature names, and adjusting core parameters of the adapter; The specific steps to adjust the core parameters of the adapter include: according to Constructing a text modal processing function ,in, is the xth part-of-speech tag, ID is the NER entity recognition model ID corresponding to the part-of-speech tag, and the corresponding entity tag set is extracted for each NER entity recognition model ID , calculate the fully connected layer dimension ; according to Dynamic sampling to extract key frame sets of video data , set the keyframe Processed as image data; according to Constructing image data processing functions , calculate the convolution kernel size ,in, is the scale factor; For audio data processing, application The silence threshold, setting the step size of the recurrent neural network ,in, is the sampling rate factor.

[0012] As a further improvement of the present invention, the specific steps of monitoring whether the existing data source is seriously missing include: Assume that the data flow of the existing data source is ; Calculate the timestamp interval, if it exists , then it is considered that the existing data source is missing. is the data spacing factor, is the normal transmission interval of the existing data source; like , it is considered that the existing data source has serious missing values, and the existing data sources with serious missing values ​​are selected and integrated to form a missing data set. Otherwise, the missing values ​​of the existing data sources are estimated by interpolation.

[0013] As a further improvement of the present invention, while binding the input feature name to the standard parameter name, the standard name tag is marked, and the existing data sources with the same standard name tag are integrated to form a pre-association set; Calculate the Euclidean space distance between the kth existing data source and the lth existing data source under the pre-association set ,when Greater than the preset maximum effective distance When , ignore the conflict calculation between the kth existing data source and the lth existing data source. Otherwise, extract the kth existing data source and the lth existing data source as a pair and integrate them to form a preliminary conflict set. The specific steps to detect whether there is a data source conflict include: Extract the existing data source with the highest sending frequency under the preliminary conflict set as the benchmark and obtain the timestamp sequence ,For the data sequence of each existing data source under the preliminary conflict set, a continuous function is generated through the cubic spline interpolation algorithm to obtain the conflict calculation set; Calculate the conflict degree for each pair of existing data sources under the conflict calculation set calculate; If the degree of conflict Greater than the dynamic conflict threshold , then this pair of existing data sources is considered to be in conflict, and each pair of existing data sources with conflict is integrated to form a conflict dataset; If the degree of conflict Greater than the maximum conflict threshold , where the maximum conflict threshold Greater than the dynamic conflict threshold , the pair of existing data sources is skipped and both existing data sources are marked as warning.

[0014] As a further improvement of the present invention, the specific steps of optimizing missing data for an existing data source with severe missing data include: reconstructing the average value of the missing data of the existing data source by the DTW reconstruction algorithm to obtain the reconstructed mean value. ; The specific steps for optimizing conflicting data for each pair of conflicting existing data sources include: Extract the most recent historical data series from existing data sources and calculate the historical data deviation of each existing data source; If the historical data deviations of the two existing data sources are both less than the corresponding deviation conflict threshold , then the two data are mean-corrected and fused; If only one existing data source has a historical data deviation greater than the corresponding deviation conflict threshold , then weight correction fusion is performed, and the value greater than the corresponding deviation conflict threshold is The existing data source is marked as warning state; If the historical data deviations of the two existing data sources are both greater than the corresponding deviation conflict threshold , then skip the data optimization process for the existing data sources and mark the two existing data sources as warning status; Calculate the historical conflict degree between the qth existing data source and the pth existing data source , adjust the deviation conflict threshold .

[0015] The present invention also proposes a system for intelligent fusion of heterogeneous multimodal data, which uses the above-mentioned method for intelligent fusion of heterogeneous multimodal data, including: a data monitoring model, which includes: Data collection module, used to collect data sources; The data analysis module is used to analyze whether the data source structure has changed with the same modality data or is new modality data; The adapter generation module generates an adapter template based on the analysis results of the data analysis module; The parsing strategy library generation module is used to dynamically expand the parsing strategy library for new modal data and send it to the adapter generation module; Multi-source data analysis module, used to detect whether there are conflicts or serious omissions in existing data sources; The data optimization module optimizes missing data for existing data sources that are severely missing based on the analysis results of the multi-source data analysis module, or optimizes conflicting data for each pair of conflicting existing data sources.

[0016] The beneficial effects of the present invention are: by constructing a data monitoring model, real-time perception of structural changes in heterogeneous multimodal data sources is achieved, and corresponding adaptive processing mechanisms are adopted according to different types of changes. When changes in the same modality data structure are detected, the system can automatically generate an adapter template to ensure the continuity and accuracy of data access; when a new modality data source is identified, the system can dynamically expand the parsing strategy library and generate a new modality adapter template to enhance the system's scalability. At the same time, the system has the ability to automatically identify and optimize multi-source data conflicts and serious missing problems, and can complete data cleaning, repair and fusion without relying on business system interfaces, significantly improving data consistency, integrity and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 It is a schematic flow diagram of the present invention. DETAILED DESCRIPTION

[0019] The accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. In order to better illustrate this embodiment, certain components of the accompanying drawings may be omitted, enlarged, or reduced in size, and do not represent the actual size of the product.

[0020] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings. The technical solution of the present invention will be further described below in conjunction with the drawings and embodiments.

[0021] Reference Figure 1 , a method for intelligent fusion of heterogeneous multimodal data, comprising the following steps: The data monitoring model is used to monitor whether the data source structure has changed. If so, it is determined whether the change is to the same modality data or to new modality data. If the change is to the same modality data, an adapter template is automatically generated. If the change is to new modality data, the parsing strategy library is dynamically expanded and a new modality adapter template is generated. Specifically, the data monitoring model is built based on the MAML meta-learning model. The core idea of ​​MAML is to find an initial parameter so that the model can quickly adapt to the new task after a small amount of gradient update. The logic of subsequent adapter generation is similar to this idea. Traditional methods often rely on manual configuration or system reconfiguration when the sensor or data modality changes, resulting in poor scalability and high maintenance workload. The present invention realizes fully automatic dynamic adaptation by automatically judging changes in the data source structure and intelligently generating adapter templates or expanding the strategy library. This greatly reduces the need for manual intervention and enables the system to seamlessly access new sensors or data sources.

[0022] As a further improvement of the present invention, the data monitoring model monitors whether the data source structure has changed. If so, the specific steps of determining whether the change is to the same modality data or to new modality data include: The data monitoring model stores multiple standard feature semantic libraries corresponding to different data types, and multiple feature baseline sets corresponding to existing data sources. The standard feature semantic libraries include multiple standard parameter names; the feature baseline sets include at least one existing parameter name and existing parameter unit corresponding to the standard parameter name. When there is a data source input, file type analysis is performed to distinguish the data type. If image data, video data, and audio data exist, metadata is extracted and regarded as the input data source. If sensor data or text data exists, the sensor data or text data is regarded as the input data source. According to the data type, the corresponding standard feature semantic library is selected, and features of the input data source are extracted using NLP natural language technology to obtain input feature names and input feature units. The input feature names are calculated one by one with the standard parameter names under the selected standard feature semantic library. If the similarity is greater than the name similarity threshold, the input feature name is bound to the standard parameter name. Specifically, file type analysis includes file header analysis, file extension detection, and MIME type detection in sequence. If there is data in a new format later, the specific detection method of file type analysis can be added accordingly. By distinguishing data types through file type analysis, the input of the data source can be preliminarily divided into modalities, so that it can be compared with standard feature names under the same data type later. For example, when an image data source is input, the file type analysis shows that the data source input is image data, and its metadata is extracted and feature extraction is performed, such as resolution. The input feature name obtained later is calculated and bound to the standard parameter name of the corresponding image data. This avoids meaningless comparison of the huge amount of standard parameter names of image data and sensor data, greatly reduces the comparison time, and can comprehensively consider the differences in data types when judging new modal data later. In addition, the existing parameter name of the present invention is to unify the input feature name into a bound standard parameter name after the input data source is connected, and mark the input feature name as an existing parameter name, so that when a new data source is input later, the corresponding existing parameter name can be directly found. In addition, while using NLP natural language processing to extract features from the input data source, a feature type recognition module can also be incorporated to categorize the features of the input data source into numerical, categorical, or temporal types. Incorporating this module allows for subsequent judgments of new modal data or changes in the same modal data to be made based on the feature type. This principle is similar to the aforementioned data type, allowing for comprehensive consideration of feature type differences when making judgments about new modal data.

[0023] At this point, it is important to remind you that metadata is data that describes the data source and is used to illustrate the basic properties of the data source. It does not directly constitute the video, image, and audio content itself, but provides key support for the understanding, management, and use of the content. Specifically, it includes resolution, frame rate, bit rate, recording time, device model, packaging format, copyright information, sampling rate, number of channels, bit depth, encoding format, and so on. The above contents are input feature names. If there is no unit, the input feature unit is recorded as 0. Different devices often have the same meaning but different names, which results in the input feature name and the standard parameter name being unable to directly correspond. Therefore, similarity calculation is required for identification. Specifically, the name is converted into a feature vector, and the cosine similarity between the two is calculated to determine whether they are synonymous sentences. For reference, the patent CN114722154B is available, and similarity calculation to identify synonyms will not be described here.

[0024] If there is no existing parameter name corresponding to the bound standard parameter name, it is determined that the input data source is new modal data; The standard feature semantic library records the standard parameter names of most data types, and the existing parameter names are the parameter names of the existing data sources currently connected to the system. If there is a lack of existing parameter names corresponding to the bound standard parameter names, it means that the current system has not yet connected to the data of this data type. In addition to the resolution, frame rate, and bit rate mentioned above, the standard parameter names also include: temperature, vibration, pressure, displacement, light, light intensity, gas concentration, humidity, current, conductivity, etc.

[0025] If each bound standard parameter name has a corresponding existing parameter name, it is considered to be homomodal data; If both exist, it means that the current system has already accessed this data type, and it can be considered as homomodal data, and the adapter can be applied directly.

[0026] Check whether each input feature unit is the same as the existing parameter unit corresponding to the same standard parameter name, and check whether each corresponding existing parameter name is in the same feature baseline set. If they are different units, mark the unit conversion label. If they are not in the same feature baseline set, mark the combination label. Otherwise, it is assumed that the data source structure has not changed. At this point, it should be noted that the data source for access here is not limited to a single sensor, a single shooting module or a recording module, but may also be directly connected to another already built multimodal data system. At this time, this multimodal data system may have a temperature sensor, a humidity sensor and a vibration sensor at the same time, and during data transmission, the connected system may directly package these sensor data and send them to the system of the present invention. However, since the system only has separate adapters for the temperature sensor, humidity sensor or vibration sensor at this time, and there is no adapter that can access these sensors at the same time, multiple adapters need to be integrated later.

[0027] Record the device code of the input data source, and no longer monitor the data source structure changes for the data source with the recorded device code.

[0028] As a further improvement of the present invention, the data monitoring model is preset with multiple pairs of unit conversion data and unit conversion factors corresponding to each pair of unit conversion data; The multiple pairs of unit conversion data specifically refer to meters and centimeters, meters and decimeters, kilometers per hour and meters per second, etc., and the unit conversion factors are 10 or 1 / 10 of meters and decimeters, and 3.6 or 0.2778 of kilometers per hour and meters per second.

[0029] The specific steps for automatically generating an adapter template if the same modality data changes include: If there is a unit conversion tag, introduce the unit conversion function in the corresponding adapter If there is a combination tag, multiple existing data sources are selected, and each bound standard parameter name has a corresponding existing parameter name in these multiple existing data sources. The adapters of the selected multiple existing data sources are integrated to form a new adapter; , , in, is the input feature unit corresponding to the unit conversion data, is the existing parameter unit corresponding to the unit conversion data, is the characteristic data of the input data source, It is the converted feature data after the feature data of the input data source is converted according to the feature unit of the existing data source. Both are unit conversion factors.

[0030] This formula is derived based on the conversion relationship of existing units. Whether n is equal to 1 or 2 depends on the relationship between the input feature unit and the existing parameter unit. If the input feature unit corresponds to meters and the existing parameter unit corresponds to decimeters, then n=1. =10, m corresponds to the unit conversion data of meter and decimeter. If the input feature unit corresponds to decimeter and the existing parameter unit corresponds to meter, then n=2. =1 / 10. Similarly, Determination and similar.

[0031] As a further improvement of the present invention, the specific steps of dynamically expanding the parsing strategy library include: For new modal data whose data type is text data, extract at least one part-of-speech tag through NLP natural language processing technology and semantic feature vector , and extract the syntax tree depth and word frequency distribution parameters , integrated to form a text feature set ; Extracting semantic feature vectors , is used to map words into a high-dimensional vector space, which can capture the semantic similarity between words. By analyzing the depth of the dependency tree constructed by grammatical relationships between sentence components, we can help understand the sentence structure and the relationship between components. , counting the frequency of occurrence of each word helps to identify the subject, keywords and other information of the text. By parsing these contents, the text data can be pre-processed in a unified manner to facilitate subsequent integration and use. It should be noted at this time that since the signal output by the sensor data is generally a numerical value + unit, or a structured text, the sensor data is regarded as text data in the present invention. If a small part of the sensor data is transmitted by other means, certain pre-processing is required. Because there are many types and models of sensors currently in the world, the present invention will not go into details one by one.

[0032] For new modal data whose data type is image data or video data, convolutional feature maps are extracted through ResNet-50 , obtained by global pooling , and extract HSV histogram statistics and edge density , integrated to form ; Specifically, for video data, a few frames can be randomly selected in each second for processing to extract these features, covering the key attributes of data semantics, color and contour. More specifically, the convolutional feature map is extracted through ResNet-50. The convolution layer captures local to global features, and global pooling retains spatial information and compresses it into a vector, thereby extracting the semantic information of the image. The HSV histogram statistics convert RGB to HSV space, count the histogram distribution of each channel, resist the interference of lighting changes, and then quantify the hue, saturation and brightness of the picture. Edge density Edges are detected using operators such as Canny, and the ratio of edge pixels per unit area is calculated.

[0033] For new modal data whose data type is audio data, extract the MFCC coefficient matrix through Librosa , where n is the number of MFCC coefficients and T is the number of time frames, the mean is calculated , standard deviation and spectrum centroid , integrated to form ; These three features represent the energy distribution, fluctuation characteristics and spectrum center of the audio, respectively, and can resist environmental noise interference and compress the data volume.

[0034] Integration to form combined features ; This facilitates the subsequent generation of parsing strategy units and generates new modal adapter templates based on the parsing strategy units.

[0035] according to Generates a parsing strategy unit.

[0036] As a further improvement of the present invention, according to The specific steps to generate a parsing strategy unit include: For the data in this article, according to Different part-of-speech tags in the NER entity recognition model are processed to obtain the part-of-speech tagging rules. ; Dynamically matching the NER model with part-of-speech tags can improve the accuracy of entity recognition. Part-of-speech tags include nouns, verbs, etc.

[0037] For image and video data, define the ROI detection threshold To determine the key areas of the image, ,in, The maximum edge strength value is detected by edge detection algorithm. is the edge detection factor; it locks the key area through the edge density threshold and reduces 90% of the redundant data processing. Specifically, This value is used to adjust the sensitivity of the ROI. A higher value tends to capture the most significant edges, making it suitable for scenes with simpler structures. A value closer to 0.5 tends to capture more edges, making it suitable for complex scenes. For example, to detect whether a circuit board is bent or dented, increase this value accordingly, such as 0.85, to avoid capturing non-critical areas unrelated to the circuit board.

[0038] For audio data, set the silence segment judgment threshold Determine the silent part in the audio and delete it. Specifically, ,in, is a constraint value; thereby extracting valid audio and improving the ability of subsequent analysis. It is to prevent A value less than 0 will result in an error or deletion of all audio. Specifically, is 0.1. Subtract twice the standard deviation It follows the two standard deviations law, and 95% of the values ​​will fall within this range. In factory scenarios, it can eliminate environmental noise, such as the vibration of operating machinery or the sound of wind.

[0039] Integrate to form combined parsing rules , through the hash function Mapped to the corresponding combination parsing rules in the hash table location.

[0040] Conveniently combine features Quickly locate the combined parsing rules .

[0041] As a further improvement of the present invention, the specific steps of generating a new modality adapter template include: selecting an adapter of an existing data source having the most correspondence between existing parameter names and input feature names, and adjusting core parameters of the adapter; Since the adapter consists of multiple aspects and multiple links, the core of the present invention lies in multi-data fusion processing. Most of the existing heterogeneous multimodal data processing is processed through the adapter. Therefore, the present invention will not elaborate on other parts of the adapter here.

[0042] The specific steps to adjust the core parameters of the adapter include: according to Constructing a text modal processing function ,in, is the xth part-of-speech tag, ID is the NER entity recognition model ID corresponding to the part-of-speech tag, and the corresponding entity tag set is extracted for each NER entity recognition model ID , calculate the fully connected layer dimension ,in, is the empirical coefficient, x represents the xth NER model; Used to quantify the semantic complexity of the current text data. The more it is, the stronger the representation capability of the fully connected layer is needed, and the dimension of the fully connected layer needs to be increased. Specifically, Can be determined directly through string calculation, empirical coefficient The default value is 1. If the model is too simple, you can increase the empirical coefficient appropriately. On the contrary, the empirical coefficient can be reduced The value of .

[0043] according to Dynamic sampling to extract key frame sets from video data ,in, is the edge strength of the t-th frame, and the key frame set Processed as image data; Since any change in motion in a video, such as object movement or camera module movement, will cause edge changes, filtering out frames with substandard edge strength can not only compress the data volume but also facilitate subsequent analysis.

[0044] according to Constructing image data processing functions ,in, , yes , calculate the convolution kernel size ,in, is the scale factor; Large convolution kernels are mainly used to capture macro defects with fewer scans, while small convolution kernels are used to focus on micro anomalies but with more scans. Different shooting modules have different scanning modes. Therefore, through Calculating the convolution kernel size can balance defect detection accuracy and computational efficiency. Specifically, the scale factor It can be obtained by dividing the size of the smallest defect of the monitored object by the resolution. For example, if the smallest defect of the monitored object is a 3cm crack, the imaging resolution of the camera module is 0.3cm / px. Then it is 10. Of course, weight coefficients can also be introduced in this calculation process. The determination of the minimum defect of the inspection object requires manual determination, so it can be set to 10 by default, so that the input data source can automatically generate an adapter. Subsequent manual adjustments can still achieve the technical effect of automatically generating an adapter.

[0045] For audio data processing, application The silence threshold, setting the step size of the recurrent neural network ,in, is the sampling rate coefficient. When the silence threshold When it is high, it means that the background noise is large and the effective audio is mainly concentrated in high-energy segments, while the silence threshold is When the value is low, it means the environment is quiet and the effective audio distribution is more dispersed. If a fixed step size is set, it will not be able to adapt to the dynamic changes in energy distribution. For example, in a high-noise environment, it is usually necessary to capture short-term fault sounds, such as equipment explosions. Therefore, this formula can be used to make the silence threshold When it is higher, shorten the step length , so that RNN scans densely and avoids missing instantaneous high-pitched faults. On the contrary, extending the step size , saving resources.

[0046] Specifically, Can be set to 0.002, at the silence threshold When it is 0.9, =2.2ms; at the silence threshold When it is 0.3, =6.6ms.

[0047] Monitor existing data sources for conflicts or serious omissions. If there are serious omissions in existing data sources, optimize the missing data for the seriously missing existing data sources. If there are conflicts in existing data sources, optimize the conflicting data for each pair of conflicting existing data sources.

[0048] It's important to note that the conflicting existing data sources mentioned here are based on existing data for the same detection object. For example, multiple vibration sensors are installed in the same area, or multiple temperature and humidity sensors are installed in a room. Due to the strong temporal and spatial nature of image, video, and audio data, conflict calculations are generally not performed on these data. In the subsequent steps, the calculations will focus on sensor data as an example.

[0049] As a further improvement of the present invention, the specific steps of monitoring whether the existing data source is seriously missing include: Assume that the data flow of the existing data source is ,in, is the timestamp, It is d-dimensional data; Calculate the timestamp interval, if it exists , then it is considered that the existing data source is missing. is the data spacing factor, is the normal transmission interval of the existing data source; like , it is considered that the existing data source has serious missing values, and the existing data sources with serious missing values ​​are selected and integrated to form a missing data set. Otherwise, the missing values ​​of the existing data sources are estimated by interpolation.

[0050] Specifically, the data interval factor The specific value of can refer to the normal signal interruption interval, such as five seconds, then Greater than 5 seconds. The data interval factor The specific value of refers to the time it takes for the device to power off and restart + the time it takes for the normal signal to be interrupted. For example, if a device takes ten seconds to restart and the normal signal interruption time is five seconds, then the data interval More than 15 seconds.

[0051] As a further improvement of the present invention, while binding the input feature name to the standard parameter name, the standard name tag is marked, and the existing data sources with the same standard name tag are integrated to form a pre-association set; Calculate the Euclidean space distance between the kth existing data source and the lth existing data source under the pre-association set ,when Greater than the preset maximum effective distance When , ignore the conflict calculation between the kth existing data source and the lth existing data source. Otherwise, extract the kth existing data source and the lth existing data source as a pair and integrate them to form a preliminary conflict set. When multiple vibration sensors are installed in the same area and this area is large, the data detected by the existing data sources located at two diagonal points of the area may have a large spatial gap. In this case, there is no need to perform conflict calculation because the vibration source may be close to one of them, resulting in one detected vibration value being higher and the other being lower.

[0052] Furthermore, conditions can be added according to actual needs, such as introducing height distance calculation, even if Euclidean space distance Less than the preset maximum effective distance However, when there is a large gap between the height positions of two existing data sources, the conflict calculation of the two existing data sources can also be ignored.

[0053] The specific steps to detect whether there is a data source conflict include: Extract the existing data source with the highest sending frequency under the preliminary conflict set as the benchmark and obtain the timestamp sequence , for each existing data source under the preliminary conflict set , all generate continuous functions through the cubic spline interpolation algorithm to obtain the conflict calculation set: , in, represents the interpolated data of the kth existing data source in the preliminary conflict set at the unified timestamp t; This method can “unify” the timestamps of each existing data source, making it easier to calculate subsequent data.

[0054] Calculate the conflict degree for each pair of existing data sources under the conflict calculation set calculate: , in, is the degree of conflict between the p-th existing data source and the q-th existing data source, n is the length of the timestamp sequence, is the weight coefficient of the i-th time point in the same timestamp sequence, is the time-based weight value, is the latest time point in the same timestamp sequence, is the i-th time point in the same timestamp sequence; If the conflict Greater than the dynamic conflict threshold , then this pair of existing data sources is considered to be in conflict, and each pair of existing data sources with conflict is integrated to form a conflict dataset; If the degree of conflict Greater than the maximum conflict threshold , where the maximum conflict threshold Greater than the dynamic conflict threshold , the pair of existing data sources is skipped and both existing data sources are marked as warning.

[0055] Since each sensor data has inevitable processing errors, assembly errors, installation errors, etc., the detection values ​​of different sensors will be different, so a dynamic conflict threshold is set. It can avoid conflict data optimization processing for each pair of existing data sources. Specifically, the dynamic conflict threshold It is determined based on the detection accuracy of each sensor model. For example, if the accuracy of an infrared distance sensor is 0.01mm, then the dynamic conflict threshold It can be set to 0.02. For the convenience of data processing, the value of the time-based weight value can be adjusted accordingly, and the dynamic conflict threshold Set to 2. In addition, the value of the time-based weight value is based on the timestamp sequence and the dynamic conflict threshold The magnification factor is determined. For example, when the farthest time point in the timestamp sequence is 23 seconds 000 milliseconds, the latest time point is 23 seconds 200 milliseconds, and the length of the timestamp sequence is 200, the weight coefficient The average value of the denominator is 100, and the dynamic conflict threshold If the image is magnified 100 times, the time-based weight value is 10,000.

[0056] Set the maximum conflict threshold , is used to distinguish different usage environments, and the Euclidean space distance Less than the maximum effective distance For example, if the two existing data sources are temperature sensor monitoring data, one is in a refrigerated room and the other is in a sunny outdoor, and the two are installed on both sides of the same wall, in this case, because of their Euclidean distance Less than the preset maximum effective distance , and has a high degree of conflict , it will be marked as a warning state, allowing monitoring personnel to manually add tags to the two data sources, such as adding "outdoor" and "indoor" tags, and treating them equally as standard name tags. If multiple temperature sensors are subsequently installed outdoors, manually adding the "outdoor" tag will allow the existing data sources with temperature detection tags and outdoor tags to be integrated into a pre-associated set.

[0057] Maximum conflict threshold The calculation is performed based on the normal value fluctuation of the area. If the normal value fluctuation is exceeded, it can be considered that the two existing data sources are in different environments, and then marked as a warning state for subsequent manual labeling.

[0058] As a further improvement of the present invention, the specific steps of optimizing missing data for an existing data source with severe missing data include: reconstructing the average value of the missing data of the existing data source by the DTW reconstruction algorithm to obtain the reconstructed mean value. ; Specifically, the reconstruction mean is obtained by the DTW reconstruction algorithm The expression is as follows: , , in, It is in the middle of the missing period The reconstructed mean of is the number of adjacent data points, is the dynamic data weight coefficient of the i-th adjacent data point, is the data vector of the i-th adjacent data point, is the number of historical similar data segments, is the dynamic data weight coefficient of the jth historical similar data segment, is the data vector of the jth historical similar data segment, The midpoint of the missing time period The time distance to the i-th adjacent data point, is the maximum allowed approach distance, is the similarity attenuation distance coefficient, is the midpoint between the jth historical similar data segment and the missing time period DTW distance.

[0059] When there are serious omissions in the data source, reconstructing each missing data will consume a huge amount of computation, so directly calculating a mean can greatly reduce the amount of computation. The number of adjacent data points is determined according to the number of missing data points, and the number of missing data points is made as close as possible to the number of adjacent data points. For the calculation algorithm of historical similar data segments and DTW distance, refer to the patent CN113225209B. It should be noted that since there is no specific data in the missing time period, the DTW distance calculation is also performed using the data of the adjacent data points, but the time is still based on the midpoint of the missing time period. To calculate.

[0060] By calculating the adjacent data points and historical similar data segments, we can not only capture the data change trend of the adjacent data points and estimate the approximate value of the missing data, but also capture the stage-by-stage value changes. For example, the data of the temperature sensor is seriously missing in the evening, and through By capturing the trend of the temperature sensor decreasing at night and increasing during the day, it can comprehensively fit the severely missing values ​​for this segment. For the dynamic data weight coefficients of adjacent data points, since the time corresponding to the adjacent data points is adjacent to the missing time period, it has a more obvious linear change, so the dynamic data weight coefficients also adopt linear changes. For the dynamic data weight coefficients of historically similar data segments, since the DTW distance is used to calculate the degree of similarity with historically similar data segments, the lower the similarity, the more the reference value drops sharply. Therefore, the e index is set for dynamic adjustment. When the DTW distance is larger, that is, the lower the similarity, the dynamic data weight coefficient value of the historically similar data segment drops sharply.

[0061] As a specific embodiment of the present invention, a temperature sensor has data loss on the evening of April 1, 2025. The missing time period is from 18:00 to 19:00. Assuming one point every 5 minutes, the number of missing points is 12. 18:30, currently needs reconstruction The average temperature value at the time. Number of adjacent data points 3, the maximum allowed proximity distance is 30 minutes, the similarity attenuation distance coefficient is 4, and the number of historical similar data segments is 2, select the nearest point before and after the missing segment, and calculate ; .

[0062] The specific steps for optimizing conflicting data for each pair of conflicting existing data sources include: Extract the recent historical data series of existing data sources and calculate the historical data deviation of each existing data source: , , , in, is the pth existing data source at time Interpolation data and historical time The data deviation, is the qth existing data source at time Interpolation data and historical time The data deviation, A timestamp sequence belonging to an existing data source used as a benchmark , is the minimum historical data time interval of the p-th data source, is the maximum historical data time interval of the pth existing data source, is the sampling frequency of the pth existing data source, is the minimum historical data time interval of the qth existing data source, is the maximum historical data time interval of the qth existing data source, is the sampling frequency of the qth existing data source, is the historical deviation degree of the p-th existing data source, is the historical deviation degree of the qth existing data source, is a timestamp sequence The number of timestamps; For each existing data source, according to the timestamp sequence The number of timestamps and the different sampling frequencies of each existing data source determine the historical time The value range of , enables the interpolated data of the data source at the unified timestamp t to be compared with the historical data one by one. Furthermore, calculating the deviation of the historical data facilitates the subsequent assessment of whether the existing data source has anomalies.

[0063] If the historical data deviations of the two existing data sources are both less than the corresponding deviation conflict threshold , then the two data are mean-corrected and fused; If only one existing data source has a historical data deviation greater than the corresponding deviation conflict threshold , then weight correction fusion is performed, and the value greater than the corresponding deviation conflict threshold is The existing data source is marked as warning state; Specifically, the expression of weight correction fusion is as follows: , when hour, ,when hour, , in, and are dynamic weight coefficients, and are the conflict weight coefficients respectively; More specifically, when hour, The value of can be 2, The value of is 1, when hour, and The value of the exchange can reduce the historical data deviation greater than the corresponding deviation conflict threshold The trust of the existing data source is enhanced, so that the fused value tends to be closer to the existing data source with normal historical data deviation.

[0064] If the historical data deviations of the two existing data sources are both greater than the corresponding deviation conflict threshold , then skip the data optimization process for the existing data sources and mark the two existing data sources as warning status; Calculate the historical conflict degree between the qth existing data source and the pth existing data source , adjust the deviation conflict threshold , Specifically, adjust the deviation conflict threshold The way is as follows: , in, is the basic conflict threshold between the qth existing data source and the pth existing data source.

[0065] Specifically, if there is no historical conflict , then the item The default value is 0, where The default value is 1. , The determination is based on the detection frequency, precision measurement unit and fluctuation range of different types of sensors. For example, the temperature detection fluctuation of a temperature sensor within ten seconds does not exceed 5 degrees.

[0066] If a minor error suddenly occurs in an existing data source, causing a data monitoring error, the conflict level will be less than the maximum conflict threshold. and is less than the dynamic conflict threshold , then it will enter the conflict data optimization process to make it biased towards normal data. If one of the existing data sources is greater than the corresponding deviation conflict threshold This indicates that the value fluctuation of the existing data source exceeds the abnormality, and the weight correction fusion and maintenance should be performed to reduce the calculation weight of the existing data source and make it biased towards normal data. , then manual inspection and maintenance should be carried out. If the inspection is normal, it can be considered that the amplitude of the fluctuation is within the normal value, and the basic conflict threshold of the existing data source should be increased accordingly.

[0067] In addition, it should be noted that if the qth existing data source and multiple existing data sources all perform conflicting data optimization processing, they should be calculated separately and the average value should be calculated at the end.

[0068] For dynamically adjusting the deviation conflict threshold The calculation principle is that if the existing data sources have a high degree of conflict for a long time, it means that raising the deviation conflict threshold can reduce false alarms for unstable data. If the conflict level is low for a long time, it means that the two data sources are in harmony for a long time, and lowering the deviation conflict threshold can improve the fusion accuracy. Specifically, the basic conflict threshold Calculations are made based on the normal fluctuation range of the existing data source. At the same time, due to the degree of conflict It is also calculated based on the difference between two existing data sources, so its value is similar to the basic conflict threshold. approximate, The default setting is 1.

[0069] At this point, the present invention can fully realize the automatic intelligent fusion processing of new data source input or new system access, from automatically and dynamically expanding the parsing strategy library and generating new modality adapter templates, to automatically detecting data missing conflicts and performing corresponding processing. When some existing data sources are specific, manual labeling is performed manually, which greatly reduces the workload. For example, when the system has accessed a certain amount of data sources, multiple temperature sensors are installed in a cold storage warehouse, where multiple temperature sensors are installed outside the warehouse and multiple temperature sensors are also installed inside the cold storage warehouse. When the first temperature sensor is connected at random, the original temperature sensor adapter template is directly applied. Since the Euclidean space distance between the first installed temperature sensor and other temperature sensors is Far greater than the preset maximum effective distance , and therefore not included in the pre-association set. When additional temperature sensors are subsequently added, they will gradually be added to this pre-association set. When only the temperature sensors within the warehouse are installed, these temperature sensors can automatically achieve data fusion, resulting in a highly accurate ambient temperature assessment. When additional temperature sensors installed outside the warehouse are added, the temperature difference will be marked as a warning, prompting the operator to manually add tags. This will cause the outdoor temperature sensors to form their own pre-association set, while the indoor temperature sensors do not require additional tags because they do not share the same standard name tags. It is important to note that the same standard name tags refer to complete compliance; one missing or one additional tag do not belong to the same standard name tags. It is important to note that the warning state is used to remind the operator to add tags. Even without this reminder, the operator should still add tags due to the different environments of the installed sensors. For example, during a certain period of winter, the room temperature in a cold storage warehouse is similar to that outside. Installing indoor and outdoor temperature sensors during this period will result in data errors in subsequent data fusion.

[0070] As a further improvement of the present invention, a method for intelligent fusion of heterogeneous multimodal data as described above is used, including: a data monitoring model, the data monitoring model includes: Data collection module, used to collect data sources; The data analysis module is used to analyze whether the data source structure has changed with the same modality data or is new modality data; The adapter generation module generates an adapter template based on the analysis results of the data analysis module; The parsing strategy library generation module is used to dynamically expand the parsing strategy library for new modal data and send it to the adapter generation module; Multi-source data analysis module, used to detect whether there are conflicts or serious omissions in existing data sources; The data optimization module optimizes missing data for existing data sources that are severely missing based on the analysis results of the multi-source data analysis module, or optimizes conflicting data for each pair of conflicting existing data sources.

[0071] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for intelligent fusion of heterogeneous multimodal data, characterized in that: The following steps are involved: The data monitoring model is used to monitor whether the data source structure has changed. If so, it is determined whether the change is to the same modality data or to new modality data. If the change is to the same modality data, an adapter template is automatically generated. If the change is to new modality data, the parsing strategy library is dynamically expanded and a new modality adapter template is generated. Monitor existing data sources for conflicts or serious omissions. If there are serious omissions in existing data sources, optimize the missing data for the seriously missing existing data sources. If there are conflicts in existing data sources, optimize the conflicting data for each pair of conflicting existing data sources.

2. The method for intelligent fusion of heterogeneous multimodal data according to claim 1, characterized in that: The data monitoring model monitors whether the data source structure has changed. If so, it determines whether the change is to the same modality data or to new modality data. The specific steps include: The data monitoring model stores multiple standard feature semantic libraries corresponding to different data types, and multiple feature baseline sets corresponding to existing data sources. The standard feature semantic libraries include multiple standard parameter names; the feature baseline sets include at least one existing parameter name and existing parameter unit corresponding to the standard parameter name. When there is a data source input, file type analysis is performed to distinguish the data type. If image data, video data, and audio data exist, metadata is extracted and regarded as the input data source. If sensor data or text data exists, the sensor data or text data is regarded as the input data source. According to the data type, the corresponding standard feature semantic library is selected, and features of the input data source are extracted using NLP natural language technology to obtain input feature names and input feature units. The input feature names are calculated one by one with the standard parameter names under the selected standard feature semantic library. If the similarity is greater than the name similarity threshold, the input feature name is bound to the standard parameter name. If there is no existing parameter name corresponding to the bound standard parameter name, it is determined that the input data source is new modal data; If each bound standard parameter name has a corresponding existing parameter name, it is considered to be homomodal data; Check whether each input feature unit is the same as the existing parameter unit corresponding to the same standard parameter name, and check whether each corresponding existing parameter name is in the same feature baseline set. If they are different units, mark the unit conversion label. If they are not in the same feature baseline set, mark the combination label. Otherwise, it is assumed that the data source structure has not changed. Record the device code of the input data source, and no longer monitor the data source structure changes for the data source with the recorded device code.

3. The method for intelligent fusion of heterogeneous multimodal data according to claim 2, characterized in that: The data monitoring model is preset with multiple pairs of unit conversion data and unit conversion factors corresponding to each pair of unit conversion data; The specific steps for automatically generating an adapter template if the same modality data changes include: If there is a unit conversion tag, introduce the unit conversion function in the corresponding adapter If there is a combination tag, multiple existing data sources are selected, and each bound standard parameter name has a corresponding existing parameter name in these multiple existing data sources. The adapters of the selected multiple existing data sources are integrated to form a new adapter.

4. The method for intelligent fusion of heterogeneous multimodal data according to claim 2, characterized in that: The specific steps for dynamically expanding the parsing strategy library include: For new modal data whose data type is text data, extract at least one part-of-speech tag through NLP natural language processing technology and semantic feature vector , and extract the syntax tree depth and word frequency distribution parameters , integrated to form a text feature set ; For new modal data whose data type is image data or video data, convolutional feature maps are extracted through ResNet-50 , obtained by global pooling , and extract HSV histogram statistics and edge density , integrated to form ; For new modal data whose data type is audio data, extract the MFCC coefficient matrix through Librosa , where n is the number of MFCC coefficients and T is the number of time frames, the mean is calculated , standard deviation and spectrum centroid , integrated to form ; Integration to form combined features ; according to Generates a parsing strategy unit.

5. The method for intelligent fusion of heterogeneous multimodal data according to claim 4, characterized in that: according to The specific steps to generate a parsing strategy unit include: For the data in this article, according to Different part-of-speech tags in the NER entity recognition model are processed to obtain the part-of-speech tagging rules. ; For image and video data, define the ROI detection threshold To identify key areas of the image; For audio data, set the silence segment judgment threshold Determine the silent part in the audio and delete it; Integrate to form combined parsing rules , through the hash function Mapped to the corresponding combination parsing rules in the hash table location.

6. The method for intelligent fusion of heterogeneous multimodal data according to claim 5, characterized in that: The specific steps of generating a new modality adapter template include: selecting an adapter for an existing data source with the most correspondence between existing parameter names and input feature names, and adjusting the core parameters of the adapter; The specific steps to adjust the core parameters of the adapter include: according to Constructing a text modal processing function ,in, is the xth part-of-speech tag, ID is the NER entity recognition model ID corresponding to the part-of-speech tag, and the corresponding entity tag set is extracted for each NER entity recognition model ID , calculate the fully connected layer dimension ; according to Dynamic sampling to extract key frame sets from video data , set the keyframe Processed as image data; according to Constructing image data processing functions , calculate the convolution kernel size ,in, is the scale factor; For audio data processing, application The silence threshold, setting the step size of the recurrent neural network ,in, is the sampling rate factor.

7. The method for intelligent fusion of heterogeneous multimodal data according to claim 2, characterized in that: Specific steps to monitor whether existing data sources are seriously missing include: Assume that the data flow of the existing data source is ; Calculate the timestamp interval, if it exists , then it is considered that the existing data source is missing. is the data spacing factor, is the normal transmission interval of the existing data source; like , it is considered that the existing data source has serious missing values, and the existing data sources with serious missing values ​​are selected and integrated to form a missing data set. Otherwise, the missing values ​​of the existing data sources are estimated by interpolation.

8. The method for intelligent fusion of heterogeneous multimodal data according to claim 7, characterized in that: While binding the input feature name to the standard parameter name, the standard name label is marked, and the existing data sources with the same standard name label are integrated to form a pre-association set; Calculate the Euclidean space distance between the kth existing data source and the lth existing data source under the pre-association set ,when Greater than the preset maximum effective distance When , ignore the conflict calculation between the kth existing data source and the lth existing data source. Otherwise, extract the kth existing data source and the lth existing data source as a pair and integrate them to form a preliminary conflict set. The specific steps to detect whether there is a data source conflict include: Extract the existing data source with the highest sending frequency under the preliminary conflict set as the benchmark and obtain the timestamp sequence ,For each existing data source under the preliminary conflict set, a continuous function is generated by using the cubic spline interpolation algorithm to obtain the conflict calculation set; Calculate the conflict degree for each pair of existing data sources under the conflict calculation set calculate; If the degree of conflict Greater than the dynamic conflict threshold , then this pair of existing data sources is considered to be in conflict, and each pair of existing data sources with conflict is integrated to form a conflict dataset; If the degree of conflict Greater than the maximum conflict threshold , where the maximum conflict threshold Greater than the dynamic conflict threshold , the pair of existing data sources is skipped and both existing data sources are marked as warning.

9. The method for intelligent fusion of heterogeneous multimodal data according to claim 8, characterized in that: The specific steps for optimizing missing data for existing data sources with severe missing data include: reconstructing the average value of the missing data of the existing data source through the DTW reconstruction algorithm to obtain the reconstructed mean value. ; The specific steps for optimizing conflicting data for each pair of conflicting existing data sources include: Extract the most recent historical data series from existing data sources and calculate the historical data deviation of each existing data source; If the historical data deviations of the two existing data sources are both less than the corresponding deviation conflict threshold , then the two data are mean-corrected and fused; If only one existing data source has a historical data deviation greater than the corresponding deviation conflict threshold , then weight correction fusion is performed, and the value greater than the corresponding deviation conflict threshold is The existing data source is marked as warning state; If the historical data deviations of the two existing data sources are both greater than the corresponding deviation conflict threshold , then skip the data optimization process for the existing data sources and mark the two existing data sources as warning status; Calculate the historical conflict degree between the qth existing data source and the pth existing data source , adjust the deviation conflict threshold .

10. A system for intelligent fusion of heterogeneous multimodal data, characterized in that: A method for intelligent fusion of heterogeneous multimodal data according to any one of claims 1 to 9, comprising: a data monitoring model, the data monitoring model comprising: Data collection module, used to collect data sources; The data analysis module is used to analyze whether the data source structure has changed with the same modality data or is new modality data; The adapter generation module generates an adapter template based on the analysis results of the data analysis module; The parsing strategy library generation module is used to dynamically expand the parsing strategy library for new modal data and send it to the adapter generation module; Multi-source data analysis module, used to detect whether there are conflicts or serious omissions in existing data sources; The data optimization module optimizes missing data for existing data sources that are severely missing based on the analysis results of the multi-source data analysis module, or optimizes conflicting data for each pair of conflicting existing data sources.

Images