A Multi-Source Data Fusion Method and System Based on Internet of Things Technology

Abstract

The present invention discloses a multi-source data fusion method and system based on Internet of Things technology, which relates to the field of electronic digital processing technology. The method includes: accessing multi-source heterogeneous data in the Internet of Things environment into the fusion system; the fusion system performs pre-fusion adaptation processing on the accessed data to obtain data stream A; cross-modal extracting feature vectors from data stream A, and fusing the feature vectors according to dynamically allocated fusion weights to obtain data stream B; quantifying the fusion error of data stream B, and further adjusting the adaptation parameters and fusion weights according to the fusion error to form a system optimization closed loop; using the system optimization closed loop to optimize the fusion system to the best and then putting it into the production environment to implement real-time fusion of the accessed data. The present invention can effectively integrate multi-modal data from different sources and in different formats in the Internet of Things environment.

Description

Technical Field

[0001] The present invention relates to the technical field of digital signal processing, and in particular, to a multi-source data fusion method and system based on Internet of Things technology. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, various intelligent terminal devices, sensors, and information systems continuously generate a large amount of heterogeneous data. These data sources are extensive, including multiple fields such as environmental monitoring, industrial equipment, intelligent transportation, and smart cities, and have characteristics such as multi-modal, high-dimensional, and strong real-time. How to efficiently integrate and analyze these multi-source data to provide accurate decision-making support has become a key challenge in IoT applications. Traditional data fusion methods are mainly based on single data sources or simple data aggregation strategies, and it is difficult to effectively handle problems such as spatio-temporal heterogeneity, data redundancy, and noise interference of multi-source data in the IoT environment. In addition, existing technologies often lack dynamic adaptability and cannot achieve real-time and efficient data fusion in complex and changing IoT scenarios, resulting in low information utilization rate and insufficient accuracy of analysis results. Therefore, there is an urgent need for a multi-source data fusion method based on IoT technology that can achieve efficient and intelligent data integration and processing, improve the ability to mine data value, and provide reliable technical support for intelligent applications. Summary of the Invention

[0003] The present invention provides a multi-source data fusion method based on Internet of Things technology, including:

[0004] Step1: Access multi-source heterogeneous data in the IoT environment into the fusion system;

[0005] Step2: The fusion system performs pre-fusion adaptation processing on the accessed data to obtain data stream A;

[0006] Step3: Cross-modally extract feature vectors from data stream A, and fuse the feature vectors according to dynamically allocated fusion weights to obtain data stream B;

[0007] Step4: Quantify the fusion error of data stream B, and further adjust the adaptation parameters in Step2 and the fusion weights in Step3 according to the fusion error to form a system optimization closed loop;

[0008] Step5: Use the system optimization closed loop to optimize the fusion system to the best and then put it into the production environment to implement real-time fusion of the accessed data.

[0009] In the multi-source data fusion method based on Internet of Things technology as described above, the fusion system performs pre-fusion adaptation processing on the accessed data, which is specifically divided into the following sub-steps:

[0010] Determine whether the pre-fusion adaptation processor for this time is executed for the first time;

[0011] If so, initialize the adaptation parameters of the pre-fusion adaptation processor based on the data characteristics in the current Internet of Things environment;

[0012] If not, fine-tune the adaptation parameters in the pre-fusion adapter based on the fusion error;

[0013] Input the standardized access data into the pre-fusion adapter to obtain data stream A.

[0014] A multi-source data fusion method based on Internet of Things technology as described above, wherein to improve the data processing rate, both the pre-fusion adaptation and fusion of access data adopt streaming processing.

[0015] A multi-source data fusion method based on Internet of Things technology as described above, wherein extract the feature vectors in data stream A across modalities, and fuse the feature vectors according to the dynamically allocated fusion weights to obtain data stream B, which is specifically divided into the following sub-steps:

[0016] Use the pre-trained cross-modal encoder to extract the feature vectors in data stream A;

[0017] Determine whether the fusion of feature vectors for this time is executed for the first time;

[0018] If so, initialize the fusion weights of each feature vector based on the principle of average distribution;

[0019] If not, fine-tune the fusion weights of each feature vector based on the fusion error;

[0020] Fuse the extracted feature vectors with the allocated fusion weights.

[0021] A multi-source data fusion method based on Internet of Things technology as described above, wherein quantify the fusion error of data stream B, which is specifically divided into the following sub-steps:

[0022] Create a fusion error calculation formula by combining the downstream task requirements and the data characteristics before and after fusion;

[0023] Use the created fusion error calculation formula to calculate the fusion error of data stream B.

[0024] The present invention also provides a multi-source data fusion system based on Internet of Things technology, including: a data access module, a pre-fusion adaptation processing module, a feature extraction and fusion module, a fusion error feedback module, and a system production module;

[0025] The data access module is used to access multi-source heterogeneous data in the Internet of Things environment into the fusion system;

[0026] The pre - fusion adaptation processing module is used to perform pre - fusion adaptation processing on the accessed data to obtain data stream A;

[0027] The feature extraction and fusion module is used to extract feature vectors from data stream A across modalities and fuse the feature vectors according to the dynamically allocated fusion weights to obtain data stream B;

[0028] The fusion error feedback module is used to quantify the fusion error of data stream B and feedback the quantified fusion error to the pre - fusion adaptation processing module and the feature extraction and fusion module to form a system optimization closed - loop;

[0029] The system production module is used to put the optimized fusion system into the production environment to implement real - time fusion of the accessed data.

[0030] A multi - source data fusion system based on the Internet of Things technology as described above, wherein the pre - fusion adaptation processing module specifically includes: an adaptation parameter adjustment sub - module and a processing result output sub - module;

[0031] The adaptation parameter adjustment sub - module is used to initialize or fine - tune the adaptation parameters in the pre - fusion adapter;

[0032] The processing result output sub - module is used to input the standardized accessed data into the pre - fusion adapter to obtain data stream A.

[0033] A multi - source data fusion system based on the Internet of Things technology as described above, wherein the feature extraction and fusion module specifically includes: a feature extraction sub - module, a fusion weight dynamic allocation sub - module, and a feature fusion sub - module;

[0034] The feature extraction sub - module is used to extract feature vectors from data stream A using a pre - trained cross - modal encoder;

[0035] The fusion weight dynamic allocation sub - module is used to initialize or fine - tune the fusion weights;

[0036] The feature fusion sub - module is used to fuse the extracted feature vectors using the allocated fusion weights.

[0037] A multi - source data fusion system based on the Internet of Things technology as described above, wherein the fusion system in the production environment does not set up the modules involved in the system optimization closed - loop. The system optimization closed - loop only dynamically optimizes the fusion system in the test environment. The modules involved in the system optimization closed - loop include: an adaptation parameter adjustment sub - module, a fusion weight dynamic allocation sub - module, and a fusion error feedback module; after the fusion system is put into production, the test environment still needs to regularly access the data in the production environment to dynamically optimize the current fusion system and then migrate it back to the production environment after optimization.

[0038] The beneficial effects achieved by the present invention are as follows: effectively integrating multi-modal data from different sources and in different formats in the Internet of Things environment (such as sensor data, images, texts, etc.), significantly improving data integrity and consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

[0040] Figure 1 It is a flowchart of a multi-source data fusion method based on Internet of Things technology provided in Embodiment 1 of this application;

[0041] Figure 2 It is a schematic diagram of a multi-source data fusion system based on Internet of Things technology provided in Embodiment 2 of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present invention.

[0043] Embodiment 1

[0044] As Figure 1 shown, Embodiment 1 of this application provides a multi-source data fusion method based on Internet of Things technology, including:

[0045] Step S110: Access multi-source heterogeneous data in the Internet of Things environment into the fusion system;

[0046] There are various networking devices in the Internet of Things environment, so a protocol adapter needs to be set at the access port of the system, which can dynamically convert different protocols to adapt to different devices. This adapter supports dynamic parsing and standardized encapsulation of multiple Internet of Things protocols such as MQTT, CoAP, and HTTP, and is compatible with the access of heterogeneous devices;

[0047] The accessed data first needs to go through timestamp synchronization and spatial coordinate mapping to solve the spatio-temporal inconsistency problem of multi-source heterogeneous data.

[0048] Step S120: The fusion system performs pre-fusion adaptation processing on the accessed data to obtain data stream A;

[0049] The accessed data comes from various devices in the Internet of Things environment and has different data formats. Therefore, an adaptation process is required before fusion to ensure the quality of the fused data and improve the adaptability to downstream analysis tasks. The specific steps are as follows:

[0050] Step S121: Determine whether the pre-fusion adaptation processor is executed for the first time;

[0051] Set a global variable Flag_JZ with an initial value of 0. If Flag_JZ == 0, it means that the pre-fusion adaptation processor is executed for the first time. If Flag_JZ == 1, it means that the pre-fusion adaptation processor is not executed for the first time.

[0052] Step S122: If so, initialize the adaptation parameters of the pre-fusion adaptation processor based on the data characteristics in the current Internet of Things environment;

[0053] The initialization formula for the Internet of Things attribute adaptation parameter d is: , where represents the lowest data dimension in the accessed data stream, represents the highest data dimension in the accessed data stream, is the adaptation coefficient of the Internet of Things attribute, is the information entropy value of the output data of the device with the highest total attribute value in the current Internet of Things environment, is the information entropy value of X; the Internet of Things attribute refers to the attributes such as the functional criticality, priority, and quantity ratio of Internet-connected devices that can measure the importance of devices, and these attribute values need to be manually preset by staff when accessing devices;

[0054] The Internet of Things correlation adaptation parameter The initialization formula is: , where is the signal-to-noise ratio of X, is the adaptation coefficient of the correlation between Internet of Things devices, is the signal-to-noise ratio of the output data of device k, P is the set of devices with a correlation relationship with the device with the highest total attribute value, and L is the total number of devices included in the device set P.

[0055] The pre-fusion adaptation processor is expressed as:

[0056] , where X is the currently accessed multi-source heterogeneous data set, is the processed data set, 、 are the balance coefficients of the Internet of Things attribute adaptation parameter and the Internet of Things correlation adaptation parameter respectively, W is the projection matrix of the adapter, , N represents the number of samples in X, D represents the dimension of X, d is the Internet of Things attribute adaptation parameter, d < D, is the Internet of Things relevance adaptation parameter, is the nuclear norm is the denoising autoencoder (DAE).

[0057] Step S123: If not, then fine-tune the adaptation parameters in the pre-fusion adapter based on the fusion error;

[0058] The fine-tuning formula for the Internet of Things attribute adaptation parameter d is: , where is the fine-tuned Internet of Things attribute adaptation, is the learning rate of the pre-fusion adapter, is the quantization result of the fusion error;

[0059] Internet of Things relevance adaptation parameter The fine-tuning formula for is: , where is the fine-tuned Internet of Things relevance adaptation parameter.

[0060] Step S124: Input the standardized access data into the pre-fusion adapter to obtain data stream A;

[0061] To improve the data processing rate, the pre-fusion adaptation of the access data and subsequent fusion both adopt streaming processing, so the output data is a data stream. The data stream output here is denoted as A.

[0062] Step S130: Extract the feature vectors from data stream A across modalities and fuse the feature vectors according to the dynamically allocated fusion weights to obtain data stream B;

[0063] Extract the feature vectors of data stream A across modalities, and then perform dynamic fusion on the extracted feature vectors to output data stream B, which is specifically divided into the following sub-steps:

[0064] Step S131: Use the pre-trained cross-modal encoder to extract the feature vectors from data stream A;

[0065] The cross-modal encoder used in this embodiment is taken from the CLIP series. Other existing cross-modal encoders such as UNITER or LXMERT can also be selected as needed, which is not limited here. The extracted feature vectors are stored in dataset C.

[0066] Step S132: Determine whether the current feature vector fusion is the first execution;

[0067] Set a global variable Flag_M with an initial value of 0. If Flag_M == 0, it means that the current feature vector fusion step is executed for the first time. If Flag_M == 1, it means that the current feature vector fusion step is not executed for the first time.

[0068] Step S133: If so, initialize the fusion weights of each feature vector based on the principle of equal distribution;

[0069] That is, the fusion weight of each feature vector is 1 / n, where n is the total number of feature vectors.

[0070] Step S134: If not, fine-tune the fusion weights of each feature vector based on the fusion error;

[0071] The fine-tuning formula for the fusion weight is expressed as: , where is the fusion weight of the i-th feature vector after fine-tuning, is the learning rate of the downstream model (i.e., the downstream model that needs to perform analysis tasks with the fused data), E is the fusion error, is a function that maps the weights to the probability simplex, , is the input parameter of the function, is the mapping vector, is the set of mapping vectors, and for each must satisfy these two conditions, is the t-th component in

[0072] Step S135: Use the allocated fusion weights to fuse the extracted feature vectors;

[0073] Use the allocated fusion weights to perform weighted summation on each feature vector in the dataset C to obtain the fused feature vector, and the data stream formed by the fused feature vector is denoted as B.

[0074] Step S140: Quantify the fusion error of the data stream B, and further adjust the adaptation parameters in step 120 and the fusion weights in step 130 to form a system optimization closed-loop;

[0075] To accurately evaluate the quality of the fusion result, it is necessary to comprehensively quantify the fusion error of the data stream B by combining the downstream task requirements and data characteristics, which is specifically divided into the following sub-steps:

[0076] Step S141: Create a fusion error calculation formula by combining the downstream task requirements and the data characteristics before and after fusion;

[0077] The fused data stream B should retain the rich information in data stream A while ensuring the smooth execution of downstream tasks. The calculation formula for the fusion error E designed based on the above requirements is expressed as: , where , , are the evaluation weights of the information richness error, task execution error, and uncertainty error respectively, represents the information entropy value of data stream A, represents the j-th data item in data stream B 's information entropy value, represents the predicted value of the downstream model for , is the expected output value of the downstream model for , represents the uncertainty score output by the Bayesian neural network for . j takes values from 1 to m, and m is the total number of data items included in data stream B.

[0078] Step S142: Calculate the fusion error of data stream B using the created fusion error calculation formula;

[0079] Step S143: Feed back the calculated fusion error to Step S120 and Step S130;

[0080] Before feedback, the global variables Flag_JZ and Flag_M need to be set to 1, which also marks the start of the system optimization closed-loop.

[0081] Step S150: Optimize the fusion system to the best using the system optimization closed-loop and then put it into the production environment to implement real-time fusion of the accessed data;

[0082] Verify the performance of the fusion system in real time according to the fusion error. When its value no longer decreases, or there is no obvious decreasing trend, it marks that the performance of the fusion system has reached the best. Migrate the current adaptive parameters and fusion weights to the production environment to implement real-time fusion of the accessed data in the current Internet of Things environment and provide high-quality fusion data for the downstream model.

[0083] Example 2

[0084] As Figure 2 shown, Embodiment 2 of the present application provides a multi-source data fusion system based on Internet of Things technology, including: a data access module 210, a pre-fusion adaptation processing module 220, a feature extraction and fusion module 230, a fusion error feedback module 240, and a system production module 250;

[0085] The data access module 210 is used to access multi-source heterogeneous data in the Internet of Things environment into the fusion system;

[0086] The pre-fusion adaptation processing module 220 is used to perform pre-fusion adaptation processing on the accessed data to obtain data stream A; specifically including: an adaptation parameter adjustment sub-module and a processing result output sub-module;

[0087] 1. The adaptation parameter adjustment sub-module is used to initialize or fine-tune the adaptation parameters in the pre-fusion adapter;

[0088] First, set a global variable Flag_JZ with an initial value of 0. If Flag_JZ == 0, it means that the pre-fusion adapter is executed for the first time in this time. If Flag_JZ == 1, it means that the pre-fusion adapter is not executed for the first time in this time; if it is the first execution, the adaptation parameters of the pre-fusion adapter are initialized based on the data characteristics in the current Internet of Things environment; if it is not the first execution, the adaptation parameters in the pre-fusion adapter are fine-tuned based on the fusion error.

[0089] 2. The processing result output sub-module is used to input the standardized accessed data into the pre-fusion adapter to obtain data stream A;

[0090] To improve the data processing rate, the pre-fusion adaptation of the accessed data and the subsequent fusion both adopt stream processing, so the output data is all a data stream, and the data stream output here is denoted as A.

[0091] The feature extraction and fusion module 230 is used to cross-modally extract the feature vectors in data stream A and fuse the feature vectors according to the dynamically allocated fusion weights to obtain data stream B; specifically including: a feature extraction sub-module, a fusion weight dynamic allocation sub-module, and a feature fusion sub-module;

[0092] 1. The feature extraction sub-module is used to extract the feature vectors in data stream A using a pre-trained cross-modal encoder;

[0093] The cross-modal encoder adopted in this embodiment is taken from the CLIP series, and other existing cross-modal encoders can also be selected as needed, such as UNITER or LXMERT, which is not limited here. The extracted feature vectors are stored in the data set C.

[0094] 2. The fusion weight dynamic allocation sub-module is used to initialize or fine-tune the fusion weights;

[0095] Set a global variable Flag_M with an initial value of 0. If Flag_M == 0, it means that this feature vector fusion step is executed for the first time. If Flag_M == 1, it means that this feature vector fusion step is not executed for the first time. If it is the first execution, initialize the fusion weights of each feature vector based on the equal distribution principle; that is, the fusion weight of each feature vector is 1 / n, where n is the total number of feature vectors. If it is not the first execution, fine-tune the fusion weights of each feature vector based on the fusion error.

[0096] The fine-tuning formula for the fusion weight is expressed as: , where is the fusion weight of the i-th feature vector after fine-tuning, is the learning rate of the downstream model (i.e., the downstream model that needs to perform the analysis task with the fused data), E is the fusion error, is a function that maps the weight to the probability simplex, , is the input parameter of the function, is the mapping vector, is the set of mapping vectors, and for each both must satisfy the two conditions, is the t-th component in

[0097] 3. Feature fusion sub-module, which is used to fuse the extracted feature vectors with the allocated fusion weights;

[0098] Use the allocated fusion weights to perform weighted summation on each feature vector in the dataset C to obtain the fused feature vector, and the data stream formed by the fused feature vector is denoted as B.

[0099] The fusion error feedback module 240 is used to quantify the fusion error of the data stream B and feedback the quantified fusion error to the pre-fusion adaptation processing module 220 and the feature extraction and fusion module 230 to form a system optimization closed-loop;

[0100] First, create a fusion error calculation formula by combining the downstream task requirements and the data characteristics before and after fusion;

[0101] The fused data stream B should retain the rich information in the data stream A and ensure the smooth execution of the downstream task. Based on the above requirements, the fusion error E calculation formula designed is expressed as: , where , , are the evaluation weights of the information richness error, task execution error, and uncertainty error respectively. represents the information entropy value of data stream A represents the j-th item of data in data stream B of the information entropy value represents the prediction value of the downstream model for of is the expected output value of the downstream model for of represents the uncertainty score output by the Bayesian neural network for , where j takes values from 1 to m, and m is the total number of data items in data stream B.

[0102] Then, use the created fusion error calculation formula to calculate the fusion error of data stream B, and feedback the calculated fusion error to the pre-fusion adaptation processing module 220 and the feature extraction and fusion module 230; before feedback, it is also necessary to set the global variables Flag_JZ and Flag_M to 1, which also marks the start of the system optimization closed-loop.

[0103] The system production module 250 is used to put the optimized fusion system into the production environment to implement real-time fusion of the accessed data;

[0104] According to the fusion error, the performance of the fusion system is verified in real time. When its value no longer decreases, or there is no obvious decreasing trend, it means that the performance of the fusion system has reached the best. At this time, the adaptive parameters and fusion weights are migrated to the production environment to implement real-time fusion of the accessed data in the current Internet of Things environment, and provide high-quality fusion data for the downstream model.

[0105] It should be noted that the fusion system in the production environment does not set the modules involved in the system optimization closed-loop. The system optimization closed-loop only dynamically optimizes the fusion system in the test environment. The modules involved in the system optimization closed-loop include: the adaptation parameter adjustment sub-module, the fusion weight dynamic allocation sub-module, and the fusion error feedback module; after the fusion system is put into production, the test environment still needs to regularly access the data in the production environment to dynamically optimize the current fusion system, and then migrate it to the production environment again after optimization to achieve the effect of production and test isolation.

[0106] Corresponding to the above embodiment, an embodiment of the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0107] The memory is used to store one or more program instructions;

[0108] The processor is used to run one or more program instructions to execute a multi-source data fusion method based on Internet of Things technology.

[0109] Corresponding to the above embodiments, an embodiment of the present invention provides a computer-readable storage medium. The computer storage medium contains one or more program instructions, and the one or more program instructions are used to be executed by a processor to implement a multi-source data fusion method based on Internet of Things technology.

[0110] An embodiment of the present invention discloses a computer-readable storage medium. Computer program instructions are stored in the computer-readable storage medium. When the computer program instructions run on a computer, the computer is enabled to execute the above-mentioned multi-source data fusion method based on Internet of Things technology.

[0111] In an embodiment of the present invention, the processor may be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0112] It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and combines its hardware to complete the steps of the above method.

[0113] The storage medium may be a memory, for example, it may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory.

[0114] Among them, the non-volatile memory may be a read-only memory (ROM for short), a programmable read-only memory (PROM for short), an erasable programmable read-only memory (EPROM for short), an electrically erasable programmable read-only memory (EEPROM for short), or a flash memory.

[0115] The volatile memory may be a Random Access Memory (RAM) which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0116] The storage media described in the embodiments of the present invention are intended to include but not be limited to these and any other suitable types of memory.

[0117] Those skilled in the art should be aware that, in one or more of the above examples, the functions described in the present invention can be implemented by a combination of hardware and software. When applying software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. The computer-readable medium includes computer storage media and communication media, where the communication media includes any medium that facilitates the transfer of a computer program from one place to another. The storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0118] The specific embodiments described above further elaborate on the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above description is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-source data fusion method based on Internet of Things technology, characterized in that, Including: Step 1: Access multi-source heterogeneous data in the Internet of Things environment into the fusion system; Step 2: The fusion system performs pre-fusion adaptation processing on the accessed data to obtain data stream A; The fusion system performs pre-fusion adaptation processing on the accessed data, which is specifically divided into the following sub-steps: Judge whether the current pre-fusion adaptation processor is executed for the first time; If so, initialize the adaptation parameters of the pre-fusion adaptation processor based on the data characteristics in the current Internet of Things environment; If not, fine-tune the adaptation parameters in the pre-fusion adapter based on the fusion error; Input the standardized accessed data into the pre-fusion adapter to obtain data stream A; Step 3: Cross-modally extract the feature vectors in data stream A, and fuse the feature vectors according to the dynamically allocated fusion weights to obtain data stream B; Step 4: Quantify the fusion error of data stream B, and further adjust the adaptation parameters in Step 2 and the fusion weights in Step 3 to form a system optimization closed-loop; Step 5: Use the system optimization closed-loop to optimize the fusion system to the best and then put it into the production environment to implement the real-time fusion of the accessed data.

2. The multi-source data fusion method based on Internet of Things technology according to claim 1, characterized in that, To improve the data processing rate, both the pre-fusion adaptation and fusion of the accessed data adopt streaming processing.

3. A multi-source data fusion method based on Internet of Things technology according to claim 1, characterized in that Cross-modally extract the feature vectors in data stream A, and fuse the feature vectors according to the dynamically allocated fusion weights to obtain data stream B, which is specifically divided into the following sub-steps: Use the pre-trained cross-modal encoder to extract the feature vectors in data stream A; Judge whether the current feature vector fusion is executed for the first time; If so, initialize the fusion weights of each feature vector based on the average distribution principle; If not, fine-tune the fusion weights of each feature vector based on the fusion error; Fuse the extracted feature vectors with the allocated fusion weights.

4. A multi-source data fusion method based on Internet of Things technology according to claim 1, characterized in that, Quantify the fusion error of data stream B, which is specifically divided into the following sub-steps: Create a fusion error calculation formula by combining the downstream task requirements and the data characteristics before and after fusion; Use the created fusion error calculation formula to calculate the fusion error of data stream B.

5. A multi-source data fusion system based on Internet of Things technology, characterized in that, Including: Data access module, pre-fusion adaptation processing module, feature extraction and fusion module, fusion error feedback module, system production module; The data access module is used to access multi-source heterogeneous data in the Internet of Things environment into the fusion system; The pre-fusion adaptation processing module is used to perform pre-fusion adaptation processing on the accessed data to obtain data stream A; The pre-fusion adaptation processing module specifically includes: an adaptation parameter adjustment sub-module and a processing result output sub-module; the adaptation parameter adjustment sub-module is used to initialize or fine-tune the adaptation parameters in the pre-fusion adaptation processor; the processing result output sub-module is used to input the standardized accessed data into the pre-fusion adapter to obtain data stream A; The feature extraction and fusion module is used to cross-modally extract the feature vectors in data stream A, and fuse the feature vectors according to the dynamically allocated fusion weights to obtain data stream B; The fusion error feedback module is used to quantify the fusion error of data stream B, and feedback the quantified fusion error to the pre-fusion adaptation processing module and the feature extraction and fusion module to form a system optimization closed loop. The feedback of the quantified fusion error to the pre-fusion adaptation processing module and the feature extraction and fusion module specifically includes: further adjusting the adaptation parameters in the pre-fusion adaptation processing module and the fusion weights in the feature extraction and fusion module according to the fusion error. The system production module is used to put the optimized fusion system into the production environment and implement the real-time fusion of the accessed data.

6. A multi-source data fusion system based on Internet of Things technology according to claim 5, characterized in that, The feature extraction and fusion module specifically includes: a feature extraction sub-module, a fusion weight dynamic allocation sub-module, and a feature fusion sub-module. The feature extraction sub-module is used to extract the feature vectors in data stream A using a pre-trained cross-modal encoder. The fusion weight dynamic allocation sub-module is used to initialize or fine-tune the fusion weights. The feature fusion sub-module is used to fuse the extracted feature vectors using the allocated fusion weights.

7. A multi-source data fusion system based on Internet of Things technology according to claim 5, characterized in that, The modules involved in the system optimization closed loop are not set in the fusion system in the production environment. The system optimization closed loop only dynamically optimizes the fusion system in the test environment. The modules involved in the system optimization closed loop include: an adaptation parameter adjustment sub-module, a fusion weight dynamic allocation sub-module, and a fusion error feedback module. After the fusion system is put into production, the test environment still needs to regularly access the data in the production environment to dynamically optimize the current fusion system, and then migrate it back to the production environment after the optimization is completed.

8. A computer storage medium, characterized in that, Including: At least one memory and at least one processor; The memory is used to store one or more program instructions. The processor is used to run one or more program instructions to execute a multi-source data fusion method based on Internet of Things technology as described in any one of claims 1-4.

Images