A multi-source heterogeneous data fusion storage system

Abstract

This invention provides a multi-source heterogeneous data fusion and storage system, including a control module, a data evaluation module electrically connected to the control module, a data analysis module electrically connected to the data evaluation module, a data integration module electrically connected to the data integration module, a data processing module electrically connected to the data processing module, a communication module electrically connected to the communication module, and a storage module electrically connected to the control module. This invention uses the data integration module to fuse and process data information, decompose the data information, determine its validity, and repair it. It also fuses data information based on data features, maps storage addresses, enabling the data to be queried through multiple methods and preventing storage redundancy.

Description

Technical Field

[0001] This invention relates to the field of multi-source heterogeneous data, and more specifically, to a multi-source heterogeneous data fusion storage system. Background Technology

[0002] A storage system is a system in a computer that consists of various storage devices for storing programs and data, control components, management information scheduling devices (hardware), and algorithms (software).

[0003] The storage system is one of the most important components of a computer. It provides the ability to write and read the information (programs and data) needed for the computer to function, thus realizing the computer's information storage function. Modern computer systems often employ a multi-level storage architecture consisting of registers, cache, main memory, and secondary storage.

[0004] Multi-source: This refers to data originating from multiple sources. Multi-source heterogeneous data comes from multiple data sources, including datasets collected from different database systems and devices during operation. Different data sources have different operating systems and management systems (e.g., OA, CRM, HR, MES, SCM), different data storage modes and logical structures, and different data generation times, usage locations, code protocols, etc., which results in the "multi-source" characteristic of data.

[0005] Heterogeneity: refers to the complexity of data types and forms, i.e., heterogeneity.

[0006] Multi-source heterogeneous data includes structured data, semi-structured data, and unstructured data.

[0007] Multi-source heterogeneous data fusion is the process of integrating various different data information, absorbing the characteristics of different data sources, and then extracting unified information that is better and richer than single data.

[0008] However, existing multi-source heterogeneous data fusion storage systems still have the following problems:

[0009] Existing technical solutions cannot effectively process data information or fully integrate multi-source heterogeneous data. Furthermore, the lack of preprocessing and cleaning of data during integration can easily lead to the presence of useless and erroneous information. In addition, the absence of address mapping for data storage makes data retrieval inconvenient, limiting the ability to query data in multiple ways, and misaligned storage locations can cause storage redundancy and other problems. Summary of the Invention

[0010] To overcome the above shortcomings, this invention provides a multi-source heterogeneous data fusion storage system, aiming to solve the problems of existing technologies being unable to effectively process data information, unable to fully integrate multi-source heterogeneous data information, and lacking pre-processing and cleaning of data information during integration processing, which easily leads to the presence of useless and erroneous information in the data information. Furthermore, the lack of address mapping for data storage makes data information querying inconvenient, unable to be queried in multiple ways, and misalignment of storage locations can easily cause storage redundancy.

[0011] This invention is implemented as follows:

[0012] A multi-source heterogeneous data fusion and storage system includes a control module electrically connected to a data evaluation module, a data analysis module electrically connected to a data analysis module, a data integration module electrically connected to a data processing module, and a communication module electrically connected to a multi-source heterogeneous module. The multi-source heterogeneous module is used to transmit multi-source heterogeneous data. The communication module is used to transmit data. The data processing module is used to perform calculations and filtering on the data. The data integration module is used to organically and virtually combine two or more datasets through format conversion, structural reorganization, semantic matching, scale conversion, and data fusion. The data analysis module mainly utilizes association analysis, classification and clustering, and deep learning techniques to realize data value mining. The data evaluation module is used to judge the integrated data, detect whether the data conforms to the meaning of the metadata information, and whether there are significant deviations.

[0013] The data integration module includes a data selection module, which is electrically connected to a data preprocessing module. The data preprocessing module is electrically connected to a data cleaning module, which is electrically connected to a data detection module. The data detection module is electrically connected to a data fusion module, and the data fusion module is electrically connected to a data processing module. The data selection module selects data to be fused from the received data. The data preprocessing module performs preliminary processing on the data. The data cleaning module detects "dirty data" and improves data quality through data filtering and repair. The data detection module detects missing data and supplements it. The data fusion module combines the processed data into new data. The data output module outputs the data.

[0014] The control module is electrically connected to a storage module, which is used to store data information after multi-source heterogeneous data processing and fusion. The storage module includes a storage queue module, which is electrically connected to a storage buffer module, which is electrically connected to a storage address module, which is electrically connected to a storage mapping module, and the storage mapping module is electrically connected to a local storage module, a cloud storage module, and a database module. The storage queue module is used to queue and store the data information that needs to be stored. The storage buffer module is used to cache the data information to improve the storage rate. The storage address module is used to edit the address of the stored data information. The storage mapping module is used to copy the data information for separate storage.

[0015] In one embodiment of the present invention, the control module is electrically connected to a voltage regulating module, and the voltage regulating module is electrically connected to a power supply module. The power supply module uses the mains power grid. The voltage regulating module includes a low voltage function for reducing the voltage of the mains power grid, a function for converting the AC voltage of the mains power grid into DC voltage, a function for stabilizing and regulating the reduced and converted voltage to achieve stable transmission, and a function for filtering out the AC voltage in the DC voltage.

[0016] In one embodiment of the present invention, the control module is electrically connected to an auxiliary module, which includes a display for displaying data information, control buttons for control and adjustment, indicator lights for displaying the operating status of the system, and a clock circuit for generating accurate motion. The clock circuit consists of a crystal oscillator, a crystal oscillator control chip, and a capacitor.

[0017] In one embodiment of the present invention, the multi-source heterogeneous module is the source of data information, and the information sources of the multi-source heterogeneous module include acquisition information, input information and production information. The acquisition information includes data information, video information, image information and audio information acquired by various sensors. The input information includes information in various formats and data information of various database types. The production information includes initial information and subsequently updated data information.

[0018] In one embodiment of the present invention, the data processing module includes an acquisition circuit for receiving data information, a gain circuit for amplifying the data information, a conversion circuit for format conversion of the data information, and a filtering circuit for filtering out noise from the data information. The filtering circuit is a digital filtering circuit.

[0019] In one embodiment of the present invention, the data selection module selects data information through data fusion standards and filters useful and similar data information according to the same criteria. The data preprocessing module is used to split the data information selected by the data selection module into smaller semantic fragments to facilitate subsequent processing.

[0020] In one embodiment of the present invention, the data cleaning module is used to remove useless and erroneous information from the data. First, statistical analysis methods are used to identify the erroneous values, and then the erroneous data can be cleared to achieve the purpose of data cleaning. The data detection module detects potential errors in inconsistent data based on the consistency between related data and repairs them to complete the cleaning of data from multiple data sources.

[0021] In one embodiment of the present invention, the data fusion module performs data fusion processing in three steps: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion, also known as data-level fusion, corresponds to data-level fusion, which involves directly unifying the data format, converting it into homogeneous data for unified processing and analysis, and then integrating the data in a structured manner. Feature-level fusion maps the feature fusion of data information to a subspace. Decision-level fusion uses Logistic regression to predict sentiment for text and related images respectively, and finally performs a weighted average of the two predicted probabilities to obtain the final result.

[0022] In one embodiment of the present invention, the storage queue module adopts a first-in-first-out (FIFO) and last-in-last-out (LIFO) arrangement mode, and the FIFO data information is first stored in the storage buffer module, so that the data information can be stored quickly and the storage efficiency can be improved. The storage address module is used to edit the storage address of the data information. The storage address is the number of the storage unit in the memory where the data information is stored. The memory is composed of a large number of storage units, and each unit needs to be distinguished by a number: number = address.

[0023] In one embodiment of the present invention, the storage mapping module assigns different virtual-to-physical address translation mappings to each task. The address translation function is defined in each task. The virtual address space in one task is mapped to a part of the physical memory, while the virtual address space of another task is mapped to another area in the physical memory. That is, data information is stored in different memories through address mapping, so that the data information can be accessed by accessing any memory. Generally, the data information is stored in the database module.

[0024] The beneficial effects of this invention are:

[0025] In use, this invention integrates data information through a data integration module. During this integration process, it further processes the data through a data preprocessing module, a data cleaning module, and a data detection module. By decomposing the data, the data cleaning module quickly identifies each segment and determines its validity. The data detection module then repairs any erroneous data to maintain accuracy. Data features are used to integrate the data, and a storage mapping module maps the data to different storage addresses before sending it to various storage systems. This allows for multi-faceted querying and prevents storage redundancy. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a system structure block diagram provided by an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the data integration module provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the storage module provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example

[0032] Please see Figure 1-3This invention provides a technical solution: a multi-source heterogeneous data fusion and storage system, comprising a control module, a data evaluation module electrically connected to the control module, a data analysis module electrically connected to the data evaluation module, a data integration module electrically connected to the data analysis module, a data processing module electrically connected to the data integration module, a communication module electrically connected to the data processing module, and a multi-source heterogeneous module electrically connected to the communication module. The multi-source heterogeneous module is used to transmit multi-source heterogeneous data information. The communication module is used to transmit data information. The data processing module is used to perform calculations and filtering on the data information. The data integration module is used for the organic and virtual combination of two or more datasets through format conversion, structural reorganization, semantic matching, scale conversion, and data fusion. The data analysis module mainly utilizes association analysis, classification and clustering, and deep learning techniques to realize the value mining of data. The data evaluation module is used to judge the integrated data information, detect whether the data information conforms to the meaning of the metadata information, and whether there is a huge deviation.

[0033] The data integration module includes a data selection module, which is electrically connected to a data preprocessing module. The data preprocessing module is electrically connected to a data cleaning module, which is electrically connected to a data detection module. The data detection module is electrically connected to a data fusion module, and the data fusion module is electrically connected to a data processing module. The data selection module selects data to be fused from the received data. The data preprocessing module performs preliminary processing on the data. The data cleaning module detects "dirty data" and improves data quality through data filtering and repair. The data detection module detects missing data and supplements it. The data fusion module combines the processed data into new data. The data output module outputs the data.

[0034] The control module is electrically connected to a storage module, which is used to store data information after multi-source heterogeneous data processing and fusion. The storage module includes a storage queue module, which is electrically connected to a storage buffer module, which is electrically connected to a storage address module, which is electrically connected to a storage mapping module, and the storage mapping module is electrically connected to a local storage module, a cloud storage module, and a database module. The storage queue module is used to queue and store the data information that needs to be stored. The storage buffer module is used to cache the data information to improve the storage rate. The storage address module is used to edit the address of the stored data information. The storage mapping module is used to copy the data information for separate storage.

[0035] To ensure stable operation of the system and provide power, in one embodiment of the present invention, a voltage regulating module is electrically connected to the control module, and a power supply module is electrically connected to the voltage regulating module. The power supply module uses the mains power grid. The voltage regulating module includes functions for reducing the voltage of the mains power grid, converting the AC voltage of the mains power grid into DC voltage, stabilizing the reduced and converted voltage for stable transmission, and filtering out the AC voltage from the DC voltage.

[0036] To make the system more convenient and faster to operate and easier to use, in one embodiment of the present invention, the control module is electrically connected to an auxiliary module. The auxiliary module includes a display for displaying data information, control buttons for control and adjustment, indicator lights for displaying the operating status of the system, and a clock circuit for generating accurate motion. The clock circuit consists of a crystal oscillator, a crystal oscillator control chip, and a capacitor.

[0037] In order to achieve multi-source heterogeneous acquisition of data information and to achieve the fusion and acquisition of different data information, in one embodiment of the present invention, the multi-source heterogeneous module is the source of data information, and the information sources of the multi-source heterogeneous module include acquisition information, input information and production information. The acquisition information includes data information, video information, image information and audio information collected by various sensors. The input information includes information in various formats and data information of various database types. The production information includes initial information and subsequently updated data information.

[0038] In order to perform calculations and processing on the data and prevent data loss during transmission, in one embodiment of the present invention, the data processing module includes an acquisition circuit for receiving data, a gain circuit for amplifying the data, a conversion circuit for format conversion of the data, and a filtering circuit for filtering out noise from the data. The filtering circuit is a digital filtering circuit.

[0039] In order to select similar or integrate transmitted data information according to standards, in one embodiment of the present invention, the data selection module selects data information through data fusion standards and filters useful and similar data information according to the same criteria. The data preprocessing module is used to split the data information selected by the data selection module into smaller semantic fragments to facilitate subsequent processing.

[0040] In order to process useless and error messages in data information and repair useful error messages, in one embodiment of the present invention, the data cleaning module is used to remove useless and erroneous information from the data information. First, statistical analysis methods are used to identify the erroneous values ​​that appear, and then the erroneous data can be cleared to achieve the purpose of data cleaning. The data detection module detects potential errors in inconsistent data based on the consistency between related data and repairs them to complete the cleaning of data from multiple data sources.

[0041] To achieve data fusion processing and effective fusion based on data characteristics, in one embodiment of the present invention, the data fusion module performs data fusion processing in three steps: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion, also known as data-level fusion, corresponds to data-level fusion, which directly unifies the data format, converts it into homogeneous data for unified processing and analysis, and then integrates the data in a structured manner. Feature-level fusion maps the features of data information to a subspace. Decision-level fusion uses Logistic regression to predict sentiment for text and related images respectively, and finally calculates a weighted average of the two predicted probabilities to obtain the final result.

[0042] To achieve effective storage of data and to set addresses for easy retrieval, in one embodiment of the present invention, the storage queue module adopts a first-in-first-out (FIFO) and last-in-last-out (LIFO) arrangement. FIFO data is first stored in the storage buffer module, enabling rapid data entry and improving storage efficiency. The storage address module is used to edit the storage address of the data. The storage address is the number of the storage unit in the memory where the data is stored. The memory consists of numerous storage units, each of which needs to be identified by a number: number = address.

[0043] To enable data storage and address mapping for various storage methods, in one embodiment of the present invention, the storage mapping module assigns different virtual-to-physical address translation mappings to each task. The address translation function is defined in each task. The virtual address space in one task is mapped to a portion of physical memory, while the virtual address space of another task is mapped to another area in physical memory. That is, data information is stored in different memories through address mapping, so that the data information can be accessed by accessing any memory. Generally, the data information is stored in the database module.

[0044] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A multi-source heterogeneous data fusion and storage system, characterized in that, The system includes a control module electrically connected to a data evaluation module, which in turn is electrically connected to a data analysis module. The data analysis module is electrically connected to a data integration module, which is electrically connected to a data processing module. The data processing module is electrically connected to a communication module, which is electrically connected to a multi-source heterogeneous module. The multi-source heterogeneous module is used to transmit heterogeneous data from multiple sources. The communication module is used to transmit data. The data processing module is used to perform calculations and filtering on the data. The data integration module is used to organically and virtually combine two or more datasets through format conversion, structural reorganization, semantic matching, scale conversion, and data fusion. The data analysis module mainly utilizes association analysis, classification and clustering, and deep learning techniques to achieve data value mining. The data evaluation module is used to judge the integrated data, detect whether the data conforms to the meaning of the metadata information, and whether there are significant deviations. The control module is electrically connected to an auxiliary module, which includes a display for displaying data information, control buttons for control and adjustment, indicator lights for displaying the system's operating status, and a clock circuit for generating accurate motion. The clock circuit consists of a crystal oscillator, a crystal oscillator control chip, and a capacitor. The data integration module includes a data selection module, which is electrically connected to a data preprocessing module. The data selection module selects data information based on data fusion standards and filters useful and similar data information according to the same criteria. The data preprocessing module is used to split the data information selected by the data selection module into smaller semantic fragments to facilitate subsequent processing. The control module is electrically connected to a storage module, which is used to store data information after multi-source heterogeneous data processing and fusion. The storage module includes a storage queue module, which is electrically connected to a storage buffer module. The storage buffer module is electrically connected to a storage address module, which is electrically connected to a storage mapping module. The storage mapping module is electrically connected to a local storage module, a cloud storage module, and a database module. The storage queue module is used to queue and store the data information that needs to be stored. The storage buffer module is used to cache the data information to improve the storage rate. The storage address module is used to edit the address of the stored data information. The storage mapping module is used to copy the data information for separate storage. The storage mapping module assigns different virtual-to-physical address translation mappings to each task. The address translation function is defined in each task. The virtual address space in one task is mapped to a part of the physical memory, while the virtual address space of another task is mapped to another area in the physical memory. That is, data information is stored in different memories through address mapping, so that the data information can be accessed by accessing any memory. Generally, the data information is stored in the database module.

2. The multi-source heterogeneous data fusion storage system according to claim 1, characterized in that, The control module is electrically connected to a voltage regulating module, which is electrically connected to a power supply module. The power supply module uses the mains power grid. The voltage regulating module includes a low-voltage function for reducing the voltage of the mains power grid, a function for converting the AC voltage of the mains power grid into DC voltage, a function for stabilizing and regulating the reduced and converted voltage to achieve stable transmission, and a function for filtering out the AC voltage from the DC voltage.

3. The multi-source heterogeneous data fusion storage system according to claim 1, characterized in that, The data selection module is electrically connected to a data preprocessing module, which in turn is electrically connected to a data cleaning module. The data cleaning module is electrically connected to a data detection module, which is electrically connected to a data fusion module. The data fusion module is electrically connected to a data processing module. The data selection module selects data to be fused from the received data. The data preprocessing module performs preliminary processing on the data. The data cleaning module detects "dirty data" and improves data quality through data filtering and repair. The data detection module detects missing data and supplements it. The data fusion module combines the processed data into new data. The data output module outputs the data.

4. The multi-source heterogeneous data fusion and storage system according to claim 1, characterized in that, The multi-source heterogeneous module is the source of data information, and the information sources of the multi-source heterogeneous module include acquisition information, input information and production information. The acquisition information includes data information, video information, image information and audio information collected by various sensors. The input information includes information in various formats and data information of various database types. The production information includes initial information and subsequent updated data information.

5. The multi-source heterogeneous data fusion and storage system according to claim 1, characterized in that, The data processing module includes an acquisition circuit for receiving data information, a gain circuit for amplifying the data information, a conversion circuit for format conversion of the data information, and a filtering circuit for filtering out noise from the data information. The filtering circuit is a digital filtering circuit.

6. The multi-source heterogeneous data fusion and storage system according to claim 3, characterized in that, The data cleaning module is used to remove useless and erroneous information from the data. First, statistical analysis methods are used to identify the erroneous values ​​before the erroneous data is cleared, thus achieving the purpose of data cleaning. The data detection module detects potential errors in inconsistent data based on the consistency between related data and repairs them to complete the cleaning of data from multiple data sources.

7. The multi-source heterogeneous data fusion storage system according to claim 1, characterized in that, The data fusion module performs data fusion processing in three steps: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion is also called data-level fusion. In contrast, data-level fusion is seen as directly unifying the data format, converting it into data of the same modality for unified processing and analysis, and then integrating the data in a structured manner. Feature-level fusion maps the feature fusion of data information to a subspace. The decision-making layer uses Logistic regression to predict sentiment for both text and related images, and then performs a weighted average of the two predicted probabilities to obtain the final result.

8. The multi-source heterogeneous data fusion storage system according to claim 1, characterized in that, The storage queue module adopts a first-in-first-out (FIFO) and last-in-last-out (LIFO) arrangement mode, and the FIFO data is first stored in the storage buffer module, which enables the data to be stored quickly and improves storage efficiency. The storage address module is used to edit the storage address of the data. The storage address is the number of the storage unit in the memory where the data is stored. The memory is composed of a large number of storage units, and each unit needs to be identified by a number: number = address.

Images