Multi-source data fusion processing method and system

Abstract

The invention discloses a multi-source data fusion processing method and system. The multi-source data fusion processing method comprises the following steps: data collection: collecting latest data through data collection equipment; data classification marking: matching the data and carrying out classification marking to generate updated data; data fusion and preprocessing: developing an efficient data preprocessing method, including data cleaning, standardization and missing value processing; data visualization and dynamic association analysis: developing a dynamic data visualization tool to visually display multi-dimensional and multi-modal characteristics of data, including visualization of real-time data streams; exporting the fused data: exporting the data and processing the data to generate the fused data. According to the method, fusion of high-dimensional and multi-modal heterogeneous image data is realized, screening and judgment are carried out based on technical means such as semantic recognition and statistical clustering, cross-industry large model modeling including finance, high and new technology industry and population information characteristics is constructed, and data visualization and dynamic association are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source data fusion, and specifically provides a multi-source data fusion processing method and system. Background Art

[0002] The multi-source data fusion processing system not only relies on a single data source, but integrates multiple types of data, including information in multiple aspects such as geographical information, demographics, electricity load, environmental protection requirements, finance, and high-tech industries; the system integrates, cleans, and standardizes data from various sources through technical means to ensure data consistency and comparability, and this step usually involves processes such as data cleaning, data conversion, and data integration for subsequent analysis and decision support;

[0003] Deficiencies of the prior art: Traditional multi-source data fusion processing systems need to integrate data from multiple fields, including geographical information, historical data, socio-economic factors, etc. However, traditional technologies often fail to effectively integrate these data from different fields, making it difficult to conduct comprehensive multi-dimensional analysis;

[0004] Lack of an effective data cross-validation mechanism makes it impossible to ensure the consistency and accuracy between different data sources, which may lead to planning decisions based on inconsistent or biased data, thus affecting the accuracy and reliability of power grid planning. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-source data fusion processing method and system to solve the problems raised in the above background art.

[0006] To achieve the above purpose, the present invention provides the following technical solution: A multi-source data fusion processing method, and the multi-source data fusion processing method is as follows:

[0007] (1) Data collection: Collect the latest data through data collection devices;

[0008] (2) Data classification and marking: Match and classify the data and generate updated data through marking;

[0009] (3) Data fusion and preprocessing: Develop efficient data preprocessing methods, including data cleaning, normalization, and missing value processing;

[0010] (4) Data visualization and dynamic correlation analysis: Develop dynamic data visualization tools to intuitively display the multi-dimensional and multi-modal characteristics of data, including the visualization of real-time data streams;

[0011] (5) Export of fused data: Export the data and process it to generate fused data;

[0012] (6) Data matching and marking missing data: Corresponding the updated data with the fused data, marking and processing the data differences to generate repaired data;

[0013] (7) Integration of all-in-one machine platform and system function testing: Integrating all developed modules into a unified system platform to ensure seamless docking of all parts and smooth data transmission.

[0014] Preferably, the multi-source data fusion method further includes real-time data repair and storage: Performing data fusion processing on the fused data and the repaired data to generate result data, and storing the result data in the cloud server.

[0015] Preferably, the multi-source data fusion method further includes retrieving and displaying result data: The user uses an electronic device to retrieve the result data and display it.

[0016] Preferably, the cloud server starts encrypting and retrieving the corresponding image data, and at the same time processes this type of image data through cloud computing to generate fused data.

[0017] Preferably, the repaired data and the fused data are matched and fused according to the same region. At the same time, the repaired data replaces some of the original data in the fused data. After the fusion is completed, the data is processed to generate result data, and the result data is transmitted to the database of the cloud server in an encrypted transmission manner for storage in the form of encoding.

[0018] Preferably, after the user inputs a search value through the electronic device, the electronic device starts comparing the search value with the result data. After the comparison is completed, the corresponding result data is displayed after A / D conversion. The electronic device is one of a computer, a mobile phone or a tablet.

[0019] Preferably, a multi-source data fusion processing system, characterized in that: it includes a data collection module, a data matching module, a map missing comparison module, a repair module, a cloud server, a display module and a retrieval module;

[0020] Data collection module: Collect the latest data;

[0021] Data matching module: Matching and classifying and marking the corresponding latest data to generate updated data;

[0022] Cloud server: Exporting the corresponding map data in the database, processing it to generate fused data, receiving the result data and storing it;

[0023] Missing comparison module: Matching the updated data with the fused data and marking and processing the data differences to generate repaired data;

[0024] Repair module: Fuse the repair data with the fusion data and process to generate result data, and send the result data to the cloud server and the display module respectively;

[0025] Display module: Perform A / D conversion on the result data and display it;

[0026] Retrieval module: Retrieve the result data by inputting a search value and display it through the display module.

[0027] Preferably, a computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. It is characterized in that when the processor executes the computer program, the steps of the method described in any one of the above claims 1-6 are implemented.

[0028] Preferably, a computer-readable storage medium stores a computer program. It is characterized in that when the computer program is run by a processor, the steps of the method described in any one of the above claims 1-6 are executed.

[0029] A multi-source data fusion processing method and system proposed by the present invention has the following beneficial effects:

[0030] 1. The present invention realizes the fusion of high-dimensional and multi-modal heterogeneous image data, screens and judges based on technical means such as semantic recognition and statistical clustering, constructs a cross-industry large model including financial, high-tech industry, population information characteristics, etc., and realizes data visualization and dynamic association.

[0031] 2. The present invention constructs a training data set based on cross-domain and multi-source multi-modal data collection, and at the same time integrates the proprietary data provided by the company and the authoritative data sets publicly available on the network to construct its own database involving data ethics and value judgment. Through the visualization model, an intelligent data all-in-one machine is developed to realize the integration of data collection, analysis and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of the multi-source data fusion processing method proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] Please refer to Figure 1, the present invention provides a technical solution: a multi-source data fusion processing method, and the multi-source data fusion processing method is as follows:

[0035] (1) Data collection: Collect the latest data through a data collection device;

[0036] (2) Data classification and marking: Match and classify the data and generate updated data by marking;

[0037] (3) Data fusion and preprocessing: Develop an efficient data preprocessing method, including data cleaning, normalization, and missing value processing;

[0038] (4) Data visualization and dynamic correlation analysis: Develop a dynamic data visualization tool to intuitively display the multi-dimensional and multi-modal characteristics of the data, including the visualization of real-time data streams;

[0039] (5) Export of fused data: Export the data and process it to generate fused data;

[0040] (6) Data matching and marking of missing data: Correlate the updated data with the fused data, mark the data differences, and process them to generate repaired data;

[0041] (7) Integration of all-in-one machine platform and system function testing: Integrate all developed modules into a unified system platform to ensure seamless docking of each part and smooth data transmission.

[0042] The data collection device in step (1) collects the latest data by means of web crawlers to crawl and search public API interface websites, and the data collection device is one of a data collector or a data acquisition instrument.

[0043] Step (3) Data fusion and preprocessing: Develop an efficient data preprocessing method, including data cleaning, normalization, and missing value processing. Implement feature extraction and fusion technologies for multi-modal data (such as images, texts, sounds). Explore and apply deep learning technologies such as convolutional neural networks (CNNs) and Transformer models to process and fuse different types of data. Design and implement a data synchronization mechanism to ensure the consistency of data in the time series.

[0044] Step (4) Data visualization and dynamic correlation analysis: Develop a dynamic data visualization tool to intuitively display the multi-dimensional and multi-modal characteristics of the data, including the visualization of real-time data streams. Build a dynamic correlation analysis framework to explore the mutual influences and relationships between different data dimensions through advanced analysis technologies such as causal inference and association rule mining. Design an interactive interface to enable end-users to adjust views and analysis parameters according to their needs to deeply explore data relationships and patterns.

[0045] Step (7) Integration of the all-in-one machine platform and system function testing: Integrate all the developed modules into a unified system platform to ensure seamless docking of all parts and smooth data transmission. Conduct performance evaluation and optimization of the system, including tests on processing speed, accuracy, and scalability. Carry out user testing to collect feedback for optimizing the user interface and functions of the system.

[0046] As a preferred solution, furthermore, the multi-source data fusion method further includes real-time data repair and storage: Perform data fusion processing on the fusion data and the repair data to generate result data, and store the result data in the cloud server. The repair data starts to match and perform fusion processing with the fusion data. At the same time, the repair data replaces some of the original data in the fusion data. After the fusion is completed, the data is processed to generate result data, and the result data is transmitted to the database of the cloud server by means of encrypted transmission and stored in the form of encoding.

[0047] As a preferred solution, furthermore, the multi-source data fusion method further includes retrieving and displaying the result data: The user uses an electronic device to retrieve and display the result data.

[0048] As a preferred solution, furthermore, the step starts to encrypt and retrieve the corresponding image data through the cloud server, and at the same time, the image data is processed through cloud computing to generate fusion data.

[0049] As a preferred solution, furthermore, the repair data and the fusion data start to match and perform fusion processing according to the same region. At the same time, the repair data replaces some of the original data in the fusion data. After the fusion is completed, the data is processed to generate result data, and the result data is transmitted to the database of the cloud server by means of encrypted transmission and stored in the form of encoding.

[0050] As a preferred solution, furthermore, after the user inputs a search value through the electronic device, the electronic device starts to compare the search value with the result data. After the comparison is completed, the corresponding result data is displayed after A / D conversion. The electronic device is one of a computer, a mobile phone, or a tablet.

[0051] As a preferred solution, furthermore, a multi-source data fusion processing system is characterized in that it includes a data collection module, a data matching module, a map missing comparison module, a repair module, a cloud server, a display module, and a retrieval module;

[0052] Data collection module: Collect the latest data;

[0053] Data matching module: Match the corresponding latest data and classify and label it to generate updated data;

[0054] Cloud server: Export the corresponding map data in the database, process and generate fusion data, receive the result data, and store it;

[0055] Missing comparison module: Match the updated data with the fusion data, mark the data differences, and generate repair data;

[0056] Repair module: Perform data fusion on the repair data and the fusion data, process and generate result data, and send the result data to the cloud server and the display module respectively;

[0057] Display module: Perform A / D conversion on the result data and display it;

[0058] Retrieval module: Retrieve the result data by inputting a search value and display it through the display module.

[0059] As a preferred solution, further, a computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1-6 when executing the computer program.

[0060] As a preferred solution, further, a computer-readable storage medium stores a computer program, wherein the computer program executes the steps of the method according to any one of claims 1-6 when run by a processor.

[0061] Working principle and usage process of the present invention: For this multi-source data fusion processing method and system, during use, the data collection module starts to collect the latest data in different regions by respectively using web crawlers to crawl and search public API interface websites, which speeds up the collection speed of real-time data. At the same time, the data processing speed is optimized, the update speed of the data map is increased, the error rate of the data map is reduced, and it is convenient for people's daily travel. After the collection is completed, the two groups of data are uploaded to the data matching module. The data matching module starts to classify and match the latest data and generates updated data, and then uploads this group of data. At the same time, the cloud server starts to encrypt and retrieve the image data of the corresponding region, and at the same time processes this type of image data through cloud computing to generate fusion data and then transmits this data. The updated data and the fusion data start to be matched according to the same region, and at the same time, the differences that occur between the updated data and the fusion data that are matched according to the same region are marked. The differences include road changes, surrounding environment changes, and traffic changes. After the marking is completed, the marked data is integrated and processed to generate repair data. The repair data and the fusion data start to be matched according to the same region and are fused. At the same time, the repair data replaces some of the original data in the fusion data, reducing the error probability generated by data fusion, saving the time of staff, and improving work efficiency. After the fusion is completed, the data is processed to generate result data, and the result data is transmitted to the database of the cloud server in an encrypted transmission manner and stored in the form of encoding. After the storage is completed, when the user inputs a search value through an electronic device, the electronic device starts to compare the search value with the result data. After the comparison is completed, the corresponding result data is displayed after A / D conversion.

[0062] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-source data fusion processing method, characterized in that, The multi-source data fusion processing method is as follows: (1) Data collection: Collect the latest data through data collection devices; (2) Data classification and marking: Match and classify the data and generate updated data with markings; (3) Data fusion and preprocessing: Develop efficient data preprocessing methods, including data cleaning, normalization, and missing value handling; (4) Data visualization and dynamic correlation analysis: Develop dynamic data visualization tools to intuitively display the multi-dimensional and multi-modal characteristics of the data, including the visualization of real-time data streams; (5) Export of fused data: Export the data and process it to generate fused data; (6) Data matching and marking of missing data: Correlate the updated data with the fused data, mark and process the data differences to generate repaired data; (7) Integration of the all-in-one machine platform and system function testing: Integrate all the developed modules into a unified system platform to ensure seamless connection of all parts and smooth data transmission.

2. A multi-source data fusion processing method according to claim 1, characterized in that: The multi-source data fusion method further includes real-time data repair and storage: Perform data fusion processing on the fused data and the repaired data to generate result data, and store the result data in a cloud server.

3. A multi-source data fusion processing method according to claim 1, characterized in that: The multi-source data fusion method further includes retrieving and displaying the result data: The user uses an electronic device to retrieve the result data and display it.

4. A multi-source data fusion processing method according to claim 1, characterized in that: In step (5), the cloud server starts to encrypt and retrieve the corresponding image data, and at the same time processes this type of image data through cloud computing to generate fused data.

5. A multi-source data fusion processing method according to claim 2, characterized in that: The repaired data and the fused data start to be matched and fused according to the same region. At the same time, the repaired data replaces some of the original data in the fused data. After the fusion is completed, the data is processed to generate result data, and the result data is transmitted to the database of the cloud server in an encrypted transmission manner for storage in an encoded form.

6. A multi-source data fusion processing method according to claim 3, characterized in that: After the user inputs a search value through the electronic device, the electronic device starts to compare the search value with the result data. After the comparison is completed, the corresponding result data is displayed after A / D conversion. The electronic device is one of a computer, a mobile phone, or a tablet.

7. A multi-source data fusion processing system, characterized in that: It includes a data collection module, a data matching module, a map missing comparison module, a repair module, a cloud server, a display module, and a retrieval module; Data collection module: Collect the latest data; Data matching module: Match and classify the corresponding latest data and generate updated data with markings; Cloud server: Export the corresponding map data in the database, process it to generate fused data, receive the result data, and store it; Missing comparison module: Match the updated data with the fused data and mark and process the data differences to generate repaired data; Repair module: Perform data fusion on the repaired data and the fused data and process it to generate result data, and send the result data to the cloud server and the display module respectively; Display module: Perform A / D conversion on the result data and display it; Retrieval module: Retrieve the result data by inputting a search value and display it through the display module.

8. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1-6 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it executes the steps of the method described in any one of claims 1-6 above.

Images