Multimodal heterogeneous data source self-defined acquisition and fusion method and system

Abstract

The invention discloses a multi-modal heterogeneous data source self-defined acquisition and fusion method and system, and relates to the technical field of multi-modal data acquisition and fusion, and the method comprises the steps: formulating a universal data model based on a business demand and a data source type; according to the universal data model, developing a corresponding adapter and a parser; a data integration tool is introduced, connection and integration of heterogeneous data sources are achieved, and a data exchange protocol is formulated; real-time data acquisition is realized by using a streaming processing technology, and data are fused based on a data exchange protocol; uniform data representation, flexible data processing, seamless heterogeneous data source integration, real-time data acquisition and processing, high expandability, data quality and consistency guarantee and flexible deployment and management are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data collection and fusion, and in particular to a method and system for customizing the collection and fusion of multimodal heterogeneous data sources. Background Art

[0002] With the deepening of digital transformation, unprecedented amounts of data have been generated within enterprises. These data are not only large in volume, but also come from diverse sources and have different structures, forming the so-called "multi-source heterogeneous data". Multi-source means that the data comes from different sources, such as databases, documents, etc.; heterogeneous means that the data format and type are inconsistent, including structured, semi-structured and unstructured data. Data from different sources may include different types of databases (such as relational databases, non-relational databases), files (such as CSV, Excel files), data returned by API interfaces, etc. The data structure, storage format, access method, etc. of these data sources may be different.

[0003] At present, these multi-source heterogeneous data are mainly processed and integrated through API combined with script tools and manual processing in scenarios such as data integration, data analysis and data warehouse construction. In the process of processing multi-source heterogeneous data, a series of challenges and problems such as true data integration, data quality, and data security are faced: each business system operates independently, and data cannot be shared and circulated, resulting in low data utilization efficiency, affecting the decision-making and management of the enterprise; there are large differences in data format, data accuracy and data integrity between different systems, which directly affects the accuracy and reliability of data; because different data standards are adopted in the design and development process of each system, data has differences in format, naming and encoding; with the interconnection and interoperability of data, the risk of data leakage and privacy infringement also increases; the cost of data integration is high, including investment in hardware, software and maintenance. In summary, data integration faces challenges such as data silos, data quality, inconsistent standards, security and privacy, and high integration costs in the context of rapid technological development and diversified data sources. To solve these problems, enterprises need to establish unified data standards and specifications, promote data interconnection and interoperability, and take strict data quality control and security protection measures. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: how to realize the technology of automatically extracting and converting data from data sources of different types, structures and sources.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for customizing the collection and fusion of multimodal heterogeneous data sources, including:

[0008] Develop a common data model based on business requirements and data source types;

[0009] Develop corresponding adapters and parsers based on the common data model;

[0010] Introduce data integration tools to achieve the connection and integration of heterogeneous data sources and formulate data exchange protocols;

[0011] Use streaming processing technology to achieve real-time data collection and fuse data based on data exchange protocols.

[0012] As a preferred solution for the customized collection and fusion method of multimodal heterogeneous data sources, it:

[0013] The general data model developed based on business requirements and data source types includes:

[0014] Analyze business requirements and determine the types of data and sources required;

[0015] A common data model is designed to accommodate metadata of different data types, wherein the data model includes common fields.

[0016] As a preferred solution for the customized collection and fusion method of multimodal heterogeneous data sources, it:

[0017] The formulation of a general data model based on business requirements and data source types also includes:

[0018] Dynamically expand the data model according to the specific data source type and add specific fields to meet the needs of different data types;

[0019] Develop metadata standardization specifications and define common metadata fields.

[0020] As a preferred solution for the customized collection and fusion method of multimodal heterogeneous data sources, it:

[0021] According to the general data model, developing corresponding adapters and parsers includes:

[0022] Apply the adapter pattern to decouple the processor and data model; each data type corresponds to an adapter, and the adapter is used to map the specific data type to the general data model; design a collection system that supports the plug-in architecture to enable new data types to be connected to the system through plug-ins.

[0023] As a preferred solution for the customized collection and fusion method of multimodal heterogeneous data sources, it:

[0024] According to the general data model, developing corresponding adapters and parsers also includes:

[0025] Develop multimodal data processors and select corresponding processing logic according to different data types;

[0026] An automatic data type recognition algorithm is introduced to automatically determine the data type by analyzing the characteristics, structure and content of the data, and dynamically select the corresponding processor.

[0027] As a preferred solution for the customized collection and fusion method of multimodal heterogeneous data sources, it:

[0028] The introduction of data integration tools to achieve the connection and integration of heterogeneous data sources and the formulation of data exchange protocols include:

[0029] Select data integration tools and common data exchange protocols based on performance and data type requirements;

[0030] Use data integration tools to convert data from heterogeneous data sources into a unified protocol;

[0031] Develop adapters based on the interface specifications of the data source and map different interfaces to standard interfaces;

[0032] Introduce middleware to handle message transmission, data conversion and protocol adaptation between different systems;

[0033] For data sources that do not support standard APIs, develop universal API interfaces.

[0034] As a preferred solution for the customized collection and fusion method of multimodal heterogeneous data sources, it:

[0035] The use of streaming technology to achieve real-time data collection and the fusion of data based on the data exchange protocol include:

[0036] Introduce streaming processing technology to establish real-time data stream pipelines to achieve instant collection, processing and distribution of real-time data;

[0037] Introduce a real-time data monitoring mechanism to detect and handle abnormal situations by monitoring data flows;

[0038] Based on the data exchange protocol, the collected real-time data is fused to generate a unified data view.

[0039] In a second aspect, an embodiment of the present invention provides a multi-modal heterogeneous data source custom collection and fusion system, including:

[0040] Data model design module, used to develop common data models based on business requirements and data source types;

[0041] The adapter and parser development module is used to develop corresponding adapters and parsers based on the common data model;

[0042] Data integration and protocol formulation module, which is used to introduce data integration tools, realize the connection and integration of heterogeneous data sources, and formulate data exchange protocols;

[0043] The real-time data collection and fusion module is used to realize real-time data collection using streaming processing technology and to fuse data based on the data exchange protocol.

[0044] In a third aspect, an embodiment of the present invention provides a computing device, including:

[0045] Memory and processor;

[0046] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the one or more programs are executed by the one or more processors, the one or more processors implement the customized collection and fusion method of multimodal heterogeneous data sources as described in any embodiment of the present invention.

[0047] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, can implement the method for customized collection and fusion of multimodal heterogeneous data sources.

[0048] Beneficial effects of the present invention: The present invention formulates a universal data model to support the representation and storage of multiple data types. Develop corresponding adapters and parsers that can handle various data types. Solve the problem that data may come from different fields, formats and structures, such as text, images, audio, etc., and their collection and processing methods are different; introduce data integration tools to support the connection and integration of heterogeneous data sources. Formulate standard data exchange protocols to ensure that various data sources can be seamlessly connected. Solve the problem that data may be distributed in different heterogeneous data sources, such as databases, file systems, APIs, etc., and unified collection becomes complicated; use streaming processing technology to achieve instant collection of real-time data. Use a distributed computing architecture to expand the collection system horizontally to ensure its performance and scalability. Solve the problem that some data may need to be collected in real time, and the collection system needs to have good scalability to cope with the growing amount of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0050] Figure 1 It is an overall flow chart of the multimodal heterogeneous data source customized collection and fusion method described in the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0054] Example 1

[0055] Reference Figure 1 , which is the first embodiment of the present invention, and provides a method for customizing the collection and fusion of multimodal heterogeneous data sources, including:

[0056] S1: Develop a common data model based on business requirements and data source types;

[0057] S2: Develop corresponding adapters and parsers based on the common data model;

[0058] S3: Introduce data integration tools to achieve the connection and integration of heterogeneous data sources and formulate data exchange protocols;

[0059] S4: Use streaming technology to achieve real-time data collection and fuse data based on data exchange protocols.

[0060] It should be noted that through the above steps, this embodiment provides an efficient and flexible multi-modal heterogeneous data source customized collection and fusion method and system. The system can effectively cope with the diversity and complexity of multi-source data and meet the data collection, processing and analysis needs in different business scenarios.

[0061] Example 2

[0062] Reference Figure 1 , which is an embodiment of the present invention, provides a method for customizing the collection and fusion of multimodal heterogeneous data sources based on the previous embodiment, including:

[0063] In the embodiment of the present application, in the above step S1, based on business requirements and data source types, formulating a general data model includes:

[0064] Analyze business needs and determine the types and sources of data required (such as text, images, audio, etc.);

[0065] A common data model is designed to accommodate metadata of different data types; the data model includes common fields such as ID, timestamp, label, etc.

[0066] Based on the specific data source type, the data model is dynamically expanded and specific fields are added to meet the needs of different data types.

[0067] Develop metadata standardization specifications and define common metadata fields, such as data source, quality, life cycle and other information.

[0068] In another possible implementation, by analyzing the business process in detail, key business links and data demand points are identified to provide a clearer basis for data fusion goals.

[0069] You can also interview various stakeholders related to the project (such as business departments, IT departments, data analysis teams, etc.) to understand their specific needs and expectations and ensure the comprehensiveness and accuracy of the integration goals.

[0070] You can also conduct a risk assessment on candidate data sources, including data quality, update frequency, privacy compliance, etc., and select data sources with lower risks and higher reliability.

[0071] In the embodiment of the present application, in the above step S2, developing corresponding adapters and parsers according to the general data model includes:

[0072] Adapter development: Apply the adapter pattern to decouple the processor and data model. Each data type corresponds to an adapter, which is responsible for mapping the specific data type to the general data model; design a collection system that supports the plug-in architecture so that new data types can be connected to the system in the form of plug-ins. The plug-in registration mechanism can automatically identify and load new data processors.

[0073] Parser development: Develop multimodal data processors and select corresponding processing logic according to different data types.

[0074] Exemplarily, an image processor is used to parse image data, and a text processor is used to parse text data.

[0075] An automatic data type recognition algorithm is introduced to automatically determine the data type by analyzing the characteristics, structure and content of the data, and dynamically select the corresponding processor.

[0076] In the embodiment of the present application, the data integration tool is introduced in the above step S3 to realize the connection and integration of heterogeneous data sources, and the data exchange protocol is formulated including:

[0077] Choose appropriate data integration tools, such as Apache NiFi, Talend, Apache Camel, etc. These tools support multiple protocols and formats, and can configure data flows through a graphical interface to simplify the complexity of integrating heterogeneous data sources.

[0078] Choose a suitable general data exchange protocol, such as JSON, XML, or Protocol Buffer, based on performance and data type requirements.

[0079] Through data integration tools, data from heterogeneous data sources are converted into a unified protocol to ensure data consistency and integrability.

[0080] Develop adapters based on the interface specifications of the data source, map different interfaces to standard interfaces, and enable the system to easily connect to different data sources.

[0081] Middleware is introduced as an intermediary for the integration of heterogeneous data sources to handle message transmission, data conversion and protocol adaptation between different systems, thereby achieving seamless integration between systems.

[0082] For data sources that do not support standard APIs, a universal API interface is developed so that integration tools can interact with the data source by calling the universal API to achieve data collection and integration.

[0083] Define data formats, interface standards, error handling mechanisms, etc. to ensure that the integration of all data sources follows the same specifications and improve the consistency of integration.

[0084] In the embodiment of the present application, the streaming processing technology is used in the above step S4 to realize real-time data collection, and the data is integrated based on the data exchange protocol, including:

[0085] Apache Flink streaming technology is introduced to establish a real-time data stream pipeline to achieve instant collection, processing and distribution of real-time data.

[0086] Introduce a real-time data monitoring mechanism to promptly detect and handle abnormal situations by monitoring key indicators of data flows (such as latency, throughput, etc.). Monitoring tools and dashboards can be used to display the status of data flows.

[0087] Based on the data exchange protocol, the collected real-time data is integrated to generate a unified data view to ensure the consistency of data between different data sources.

[0088] The Apache Spark distributed computing architecture is adopted to distribute computing tasks to multiple nodes through horizontal expansion, thereby improving the processing power and scalability of the system.

[0089] Use dynamic resource scheduling mechanism to automatically adjust the allocation of system resources according to actual load and demand, and effectively respond to fluctuations and changes in data traffic.

[0090] The introduction of Kubernetes containerization technology enables rapid deployment, flexible expansion, and efficient management by containerizing the collection system components, thereby improving the elasticity and maintainability of the system.

[0091] Introduce RabbitMQ or Apache Kafka message queues to implement asynchronous communication and improve the decoupling and responsiveness of the system.

[0092] In another possible implementation, when the amount of data is large, a distributed storage and computing framework (such as Hadoop, Spark, etc.) is used to improve the efficiency and scalability of data processing and fusion.

[0093] During the data fusion process, a data quality monitoring mechanism should be established to monitor data consistency, completeness, accuracy and other indicators in real time, promptly identify and handle data quality issues, and ensure the reliability of fusion results.

[0094] Example 3

[0095] The above is a schematic scheme of the multimodal heterogeneous data source custom collection and fusion method of this embodiment. It should be noted that the technical solution of the multimodal heterogeneous data source custom collection and fusion system and the technical solution of the multimodal heterogeneous data source custom collection and fusion method above belong to the same concept. The details of the technical solution of the multimodal heterogeneous data source custom collection and fusion system not described in detail in this embodiment can be referred to the description of the technical solution of the multimodal heterogeneous data source custom collection and fusion method above.

[0096] This embodiment also provides a system based on a custom collection and fusion method of multimodal heterogeneous data sources, including:

[0097] Data model design module, used to develop common data models based on business requirements and data source types;

[0098] The adapter and parser development module is used to develop corresponding adapters and parsers based on the common data model;

[0099] Data integration and protocol formulation module, which is used to introduce data integration tools, realize the connection and integration of heterogeneous data sources, and formulate data exchange protocols;

[0100] The real-time data collection and fusion module is used to realize real-time data collection using streaming processing technology and to fuse data based on the data exchange protocol.

[0101] This embodiment further provides a computing device, which is applicable to the case of a customized collection and fusion method of multimodal heterogeneous data sources, including:

[0102] Memory and processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the customized collection and fusion method of multimodal heterogeneous data sources proposed in the above embodiment.

[0103] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for customized collection and fusion of multimodal heterogeneous data sources proposed in the above embodiment is implemented.

[0104] The storage medium proposed in this embodiment and the customized collection and fusion method of multimodal heterogeneous data sources proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0105] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for customizing the collection and fusion of multimodal heterogeneous data sources, characterized in that: include: Develop a common data model based on business requirements and data source types; Develop corresponding adapters and parsers based on the common data model; Introduce data integration tools to achieve the connection and integration of heterogeneous data sources and formulate data exchange protocols; Use streaming processing technology to achieve real-time data collection and fuse data based on data exchange protocols.

2. The method for customizing the collection and fusion of multimodal heterogeneous data sources according to claim 1, characterized in that: The general data model developed based on business requirements and data source types includes: Analyze business requirements and determine the types of data and sources required; A common data model is designed to accommodate metadata of different data types, wherein the data model includes common fields.

3. The method for self-defined collection and fusion of multimodal heterogeneous data sources according to claim 2, characterized in that: The formulation of a general data model based on business requirements and data source types also includes: Dynamically expand the data model according to the specific data source type and add specific fields to meet the needs of different data types; Develop metadata standardization specifications and define common metadata fields.

4. The method for self-defined collection and fusion of multimodal heterogeneous data sources as claimed in claim 3, characterized in that: According to the general data model, developing corresponding adapters and parsers includes: Apply the adapter pattern to decouple the processor and data model; each data type corresponds to an adapter, and the adapter is used to map the specific data type to the general data model; design a collection system that supports the plug-in architecture to enable new data types to be connected to the system through plug-ins.

5. The method for self-defined collection and fusion of multimodal heterogeneous data sources according to claim 4, characterized in that: According to the general data model, developing corresponding adapters and parsers also includes: Develop multimodal data processors and select corresponding processing logic according to different data types; An automatic data type recognition algorithm is introduced to automatically determine the data type by analyzing the characteristics, structure and content of the data, and dynamically select the corresponding processor.

6. The method for self-defined collection and fusion of multimodal heterogeneous data sources according to claim 5, characterized in that: The introduction of data integration tools to achieve the connection and integration of heterogeneous data sources and the formulation of data exchange protocols include: Select data integration tools and common data exchange protocols based on performance and data type requirements; Use data integration tools to convert data from heterogeneous data sources into a unified protocol; Develop adapters based on the interface specifications of the data source and map different interfaces to standard interfaces; Introduce middleware to handle message transmission, data conversion and protocol adaptation between different systems; For data sources that do not support standard APIs, develop universal API interfaces.

7. The method for self-defined collection and fusion of multimodal heterogeneous data sources according to claim 6, characterized in that: The use of streaming technology to achieve real-time data collection and the fusion of data based on the data exchange protocol include: Introduce streaming processing technology to establish real-time data stream pipelines to achieve instant collection, processing and distribution of real-time data; Introduce a real-time data monitoring mechanism to detect and handle abnormal situations by monitoring data flows; Based on the data exchange protocol, the collected real-time data is fused to generate a unified data view.

8. A system using the multimodal heterogeneous data source customized collection and fusion method as described in any one of claims 1 to 7, characterized in that: include: Data model design module, used to develop common data models based on business requirements and data source types; The adapter and parser development module is used to develop corresponding adapters and parsers based on the common data model; Data integration and protocol formulation module, which is used to introduce data integration tools, realize the connection and integration of heterogeneous data sources, and formulate data exchange protocols; The real-time data collection and fusion module is used to realize real-time data collection using streaming processing technology and to fuse data based on the data exchange protocol.

9. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multimodal heterogeneous data source customized collection and fusion method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method for customized collection and fusion of multimodal heterogeneous data sources as described in any one of claims 1 to 7.

Images