Multi-source data fusion method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-source data fusion method and device, electronic equipment and a storage medium, and the method comprises the steps: recognizing and connecting different types and formats of data sources according to a user input or configuration file; performing corresponding data format conversion and mapping operation according to the data source, and converting original data into a JSON document; importing the JSON document into a search data analysis engine, and creating a corresponding index or updating an existing index; and performing retrieval, aggregation and analysis operation on multi-source data through the search data analysis engine to obtain a target data result. The fusion method is based on the search data analysis engine, the characteristics of high performance, high expandability, high fault tolerance and the like of the search data analysis engine are utilized, and effective fusion of large-scale, diversified and dynamically changing data sources can be achieved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a multi-source data fusion method, device, electronic device and storage medium. Background Art

[0002] With the acceleration of my country's digitalization process and the deepening of dataization trends, more and more data need to be collected and transmitted to data centers in real time for analysis and processing. However, different data centers are not completely unified in the selection of databases and data architectures. In order to conduct real-time query and analysis and calculation of standard data of the final results, it is necessary to carry out corresponding customized development and processing for the actual data architecture, which increases time and labor costs.

[0003] ElasticSearch (hereinafter referred to as ES) is a distributed search and analysis engine based on Lucene. It has the characteristics of high performance, high scalability, high fault tolerance, etc. It can support the import and indexing of data of various types and formats, and provides powerful query and analysis functions. ES is an open source project and has been widely used in various fields, such as e-commerce, log analysis, social networks, etc. However, ES itself does not provide a method for multi-source data fusion. It is only a low-level search and analysis engine. Users need to write corresponding programs or plug-ins to realize the recognition, connection, conversion, mapping, import, indexing and update of different data sources. These operations require users to have high programming skills and a deep understanding of the internal mechanism of ES, and may involve a lot of code writing and maintenance work. Summary of the invention

[0004] In view of this, the present application aims to propose a multi-source data fusion method, device, electronic device and storage medium to solve the problem that users need to write programs or plug-ins by themselves to implement different data fusion processing operations.

[0005] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0006] In a first aspect, the present application provides a multi-source data fusion method, comprising:

[0007] Identify and connect to data sources of different types and formats based on user input or configuration files;

[0008] Perform corresponding data format conversion and mapping operations according to the data source to convert the original data into a JSON document;

[0009] Import the JSON document into the search data analysis engine and create a corresponding index or update an existing index;

[0010] The search data analysis engine is used to retrieve, aggregate and analyze multi-source data to obtain target data results.

[0011] In the second aspect, based on the same inventive concept, the present application also provides a multi-source data fusion device, comprising:

[0012] An identification and connection module is configured to identify and connect data sources of different types and formats according to user input or configuration files;

[0013] A conversion module is configured to perform corresponding data format conversion and mapping operations according to the data source to convert the original data into a JSON document;

[0014] A data import module is configured to import the JSON document into the search data analysis engine and create a corresponding index or update an existing index;

[0015] The data acquisition module retrieves, aggregates and analyzes multi-source data through the search data analysis engine to obtain target data results.

[0016] In a third aspect, based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0017] In a fourth aspect, based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0018] Compared with the prior art, the multi-source data fusion method, device, electronic device and storage medium described in this application have the following beneficial effects:

[0019] (1) The multi-source data fusion method provided by this solution is based on the search data analysis engine and utilizes the high performance, high scalability, and high fault tolerance of the search data analysis engine. It can achieve effective fusion of large-scale, diversified, and dynamically changing data sources.

[0020] (2) The multi-source data fusion method provided by this solution does not require users to perform complex data model design, data mapping rule definition, data conversion programming, etc. Users only need to input or configure the corresponding data source information to achieve automatic identification, connection, conversion, mapping, import, indexing and update synchronization of multi-source data.

[0021] (3) The multi-source data fusion method provided by this solution allows users to select appropriate data conversion and mapping methods, data import and indexing methods, data consistency and update methods based on the characteristics of different data source types, formats, frequencies, etc., to meet different data fusion requirements.

[0022] (4) This solution provides a multi-source data fusion method. Users can use the query and analysis interface provided by ES to perform unified retrieval, aggregation, and analysis operations on multi-source data without having to worry about the differences and complexity of the data sources.

[0023] (5) The multi-source data fusion method provided by this solution can effectively reduce the cost of data warehouse construction and BI customization. It does not rely on big data ecosystems such as Hadoop and HBase, avoids direct operations on the data center, and can be directly embedded in the existing digital asset catalog database, thereby achieving unified query and analysis of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 This is a flow chart of the multi-source data fusion method described in an embodiment of the present application;

[0026] Figure 2 This is a flowchart for the specific implementation of the multi-source data fusion method described in the embodiment of the present application;

[0027] Figure 3 This is a schematic diagram of the structure of the multi-source data fusion device described in an embodiment of the present application;

[0028] Figure 4 This is a schematic diagram of the hardware structure of the electronic device described in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] The purpose of data processing is to extract valuable information and knowledge from massive and diverse data to support decision-making and innovation. In order to solve the problems arising from the background technology, this application proposes a multi-source data fusion method based on ElasticSearch (search data analysis engine), which is an important branch in data processing technology, i.e., data fusion technology, which can integrate and unify data from different sources, types and formats to achieve a comprehensive and in-depth understanding and utilization of data. Data fusion is of great significance in the era of big data, because the amount of data is exploding and the sources of data are becoming more and more diverse. How to effectively fuse and analyze these multi-source data to improve the quality and value of data is a problem to be solved. This application uses ElasticSearch as the basis for data fusion and analysis, which can unify data from different sources and achieve fast, flexible and scalable fusion and analysis of data sources of different types and formats.

[0033] See also Figure 1 As shown, this embodiment provides a multi-source data fusion method, including:

[0034] Step S101: Identify and connect data sources of different types and formats according to user input or configuration files.

[0035] Specifically, in this embodiment, according to user input or configuration files, data sources of different types and formats can be identified and connected, including relational databases, non-relational databases, API interfaces, log files, etc. This step requires the use of corresponding client libraries or tools to create and test connection objects based on data source information, obtain metadata information of data sources, and classify different data formats to abstract a unified public data structure model.

[0036] Step S102: Perform corresponding data format conversion and mapping operations according to the data source to convert the original data into a JSON document.

[0037] Specifically, in this embodiment, for each data source, according to its type and format, the corresponding data format conversion and mapping operations are performed, the original data is converted into a JSON document that meets the requirements of the ES index structure, the wide field redundancy is normalized, and the corresponding metadata information is added. Among them, it is necessary to understand the basic concepts and requirements of the ES index structure, and according to different data source types, select the appropriate data conversion method and mapping method according to the defined standard data format, use the mutate filter plug-in of Logstash to realize data conversion and specify field mapping, and use the fingerprint filter plug-in of Logstash to realize data unification and deduplication filtering, etc.

[0038] Step S103: import the JSON document into the search data analysis engine, and create a corresponding index or update an existing index.

[0039] Specifically, in this embodiment, the converted and mapped JSON documents are imported into ES in batches or streams, and corresponding indexes are created or existing indexes are updated. This step requires the use of the index management interface and batch processing interface provided by ES to achieve efficient data import and index creation. At the same time, in order to adapt to different types of data sources, various data source plug-ins based on Logstash are used to achieve the connection of different types of data sources and data collection and migration.

[0040] At the same time, data consistency and updating also need to be ensured. That is, for data sources of different types and frequencies, regular incremental updates or real-time data synchronization are used to ensure that the data in ES is consistent with the data in the data source. Different strategies need to be configured in Logstash according to different data source types.

[0041] Step S104: retrieve, aggregate and analyze the multi-source data through the search data analysis engine to obtain target data results.

[0042] Specifically, in this embodiment, the query and analysis interface provided by ES is used to retrieve, aggregate and analyze multi-source data to obtain the required data results. By using the search language (DSL) or other high-level languages ​​(Elasticsearch SQL) or visualization tools (Kibana) provided by ES to build complex query and analysis statements, and process the returned results, the visualization of the final results is formed and provided to the user for display, and it can also facilitate users to perform unified global fast search.

[0043] The multi-source data fusion method described in this embodiment does not require users to perform complex data model design, data mapping rule definition, data conversion programming, etc., and only needs to input or configure the corresponding data source information to realize automatic identification, connection, conversion, mapping, import, indexing and update synchronization of multi-source data. It can effectively reduce the cost of data warehouse construction and BI customization, does not rely on big data ecosystems such as Hadoop and HBase, avoids direct operations on the data center, and can be directly nested in the existing digital asset catalog database, thereby realizing unified query and analysis of data.

[0044] Based on the above scheme, the following explanation is given in combination with specific embodiments:

[0045] like Figure 2 As shown, assume that there are the following different types of data and different types of data sources:

[0046] 1. Common type data:

[0047] (1) The relational database MySQL stores the user information table user_info (including the fields user_id, user_name, user_age, user_gender, user_address) and the order information table order_info (including the fields order_id, user_id, product_id, order_amount, order_time);

[0048] (2) The non-relational database MongoDB stores the product information set product_info (including the fields product_id, product_name, product_price, product_category, product_description);

[0049] (3) API interface, which provides a service for obtaining product review information. The input is product_id and the output is a JSON array, each element of which contains the fields review_id, user_id, product_id, review_content, review_score, and review_time;

[0050] (4) Log file, which records the user's browsing behavior log of products. Each line is a JSON object containing the fields user_id, product_id, and browse_time.

[0051] 2. Standard asset data formed after data governance:

[0052] The relational database SQLServer includes three standard asset databases: enterprise database, personnel database, and project database.

[0053] Now it is necessary to integrate these data sources into ES for query and analysis. The standard data assets formed after data governance need to be uniformly queried and analyzed externally. According to the method described in this application, the specific steps are as follows:

[0054] 1. Data source identification and connection:

[0055] 1. Install and configure ElasticSearch and related client libraries, such as Java, Python, etc., to facilitate interaction and operation with data sources.

[0056] 2. Choose the appropriate connection method and driver according to different data source types. For example, for relational databases, you can use the public configuration JDBC or ODBC and other standard interfaces to connect; for non-relational databases, you can use the corresponding client tools to connect; for API interfaces, you can use HTTP requests or other protocols to connect; for log files, you can use file reading or monitoring methods to connect.

[0057] The connection method and driver are software components used to communicate and operate with the data source. They are mainly responsible for implementing the details of the data source such as the protocol, format and syntax. Different connection methods and drivers may support different data sources and functions. It is necessary to select the appropriate connection method and driver according to the actual situation. This embodiment will not go into details.

[0058] 3. Identify and connect to the data source based on the user interface input or configuration file. For example, the user can enter or configure the following information in the interface (if no specific table information is specified, all table data in the corresponding library will be collected by default):

[0059] mysql: {host: "localhost", port: 3306, user: "root", password: "123456", database: "test", tables: ["user_info", "order_info"]}.

[0060] mongodb: {host: "localhost", port: 27017, database: "test", collection: "product_info"}.

[0061] api: {url: "http: / / example.com / review", param: "product_id"}.

[0062] log:{path:“ / home / user / log.txt”}.

[0063] sqlserver: {host: "localhost", port: 1433, user: "sa", password: "123456", database: ["company", "project" "person"]}.

[0064] Based on the information of the data source, create and test the connection object, obtain the metadata information of the data source, such as table structure, field type, index information, etc., and configure the corresponding data update strategy according to the characteristics and requirements of different data sources.

[0065] 2. Data conversion and mapping:

[0066] For each data source, perform corresponding data format conversion and mapping operations according to its type and format, and then write the corresponding JSON document into ES, for example:

[0067] mysql: For each table, convert each row into a JSON document and add metadata information. For example:

[0068] {"_index":"mysql","_type":"user_info","_id":"1","_source":{"user_id":"1","user_name":"Alice","user_age":20,"user_gender":"F","user_address":"Beijing"}}.

[0069] MongoDB: For each collection, each document is directly treated as a JSON document and metadata information is added. For example:

[0070] {"_index":"mongodb","_type":"product_info","_id":"1","_source":{"product_id":"1","product_name":"iPhone 12","product_price":6999,"product_category":"mobile phone","product_description":""}}.

[0071] For each API interface, perform corresponding data request and response operations according to its input and output, convert the response result into a JSON document, and add metadata information. For example:

[0072] {"_index":"api","_type":"review","_id":"1","_source":{"review_id":"1","user_id":"2","product_id":"1","review_content":"","review_score":5,"review_time":"2023-09-17T10:00:00Z"}}.

[0073] log: For each log file, convert each line of its record into a JSON document and add metadata information. For example:

[0074] {"_index":"log","_type":"browse","_id":"1","_source":{"user_id":"1","product_id":"2","browse_time":"2023-09-17T11:00:00Z"}}.

[0075] sqlserver: For each standard data asset library, a separate index can be created. The index is further subdivided into multiple independent types according to different basic information business tables. For each type, a common field mapping is abstracted, and standard data types and attributes are specified to facilitate subsequent supplementation and expansion of the data structure. Each piece of data is then converted into a mapped JSON document and supplemented with basic metadata information, such as _index, _type, _id, etc., to indicate the index name, type name, document ID, etc. to which the document belongs, to facilitate final data retrieval and analysis.

[0076] If different data sources contain the same basic business data, for example, both MySQL and Sqlserver databases contain basic personnel information, you need to aggregate and integrate this basic information into one ES index. This can be achieved by following the steps below:

[0077] Determine the connection information of the data source to be integrated, such as database name, address, port, user name, password, etc., and ensure that Logstash can access these data sources;

[0078] Determine the structure and fields of the data tables to be integrated, such as personnel ID card, gender, name, age, etc., and ensure that these fields have the same or similar meaning and format in different data sources.

[0079] Determine the duplicate data criteria to be filtered, such as identical ID cards and names, and ensure that these criteria have the same or similar logic and implementation in different data sources.

[0080] Write the configuration file of Logstash, use the jdbc input plug-in to query data from mysql and sqlserver, use the fingerprint filter plug-in to generate a unique fingerprint for each piece of data, and use the elasticsearch filter plug-in to query whether data with the same fingerprint already exists in ES. If it does, discard the data, if not, continue processing. Use the mutate filter plug-in to rename, delete, replace, convert the type of fields, etc. to achieve field unification. Use the elasticsearch output plug-in to output data to ES.

[0081] For example, the following simple configuration example:

[0082]

[0083]

[0084]

[0085] In the input part, the jdbc plug-in is used, and parameters such as the driver library, driver class, basic connection configuration, and query statement are specified; in order to distinguish data from different data sources, a tag is added to each input.

[0086] In the filter section, use the fingerprint plug-in and specify parameters such as method, source, and target. Here, the SHA1 algorithm is used to hash the two fields of the person's ID number and name to generate a unique fingerprint for each piece of data and store it in the fingerprint field;

[0087] Use the elasticsearch plug-in and specify parameters such as hosts, index, query_template and fields. Use a query template file to define the query statement. Use the fingerprint field to match whether the same data already exists in ES, and store the found field of the query result in the [@metadata][found] field. Use the if condition to judge the value of the [@metadata][found] field. If it is true, it means that the data is duplicated and the drop plug-in is used to discard it. If it is false, it means that the data is new and continues to be processed.

[0088] Use the mutate plugin and specify parameters such as rename, remove_field, and convert. Here, according to the mapping rules of the target index, the fields in different data sources are renamed to a unified field, unnecessary fields are deleted, and the field types are converted.

[0089] In the output part, use the elasticsearch plug-in and specify parameters such as hosts and index. The processed data is output to the person_info index in ES, thereby achieving the effect of unified integration of basic personnel information data.

[0090] 3. Data import and indexing:

[0091] Import the converted and mapped JSON documents into ES in batches or streaming, and create corresponding indexes or update existing indexes. To import multi-source data into the target index of ES, you need to use the Logstash input plug-in to obtain data from different data sources, and then use the output plug-in to send the data to ES. The following is an example of specific implementation steps:

[0092] For relational databases, you need to download the corresponding jdbc driver and put it in the same specified path, then configure and use the corresponding input jdbc plug-in to obtain data and send it to the specified index of ES;

[0093] For non-relational databases, you need to configure and install the Logstashinput plug-in dedicated to the specified non-relational database to obtain data and then send it to the specified index of ES;

[0094] For API interfaces, you can use the http_poller input plug-in to obtain data, customize the response result parsing strategy according to actual needs, and then send the data to the specified index of ES;

[0095] For log files, you can use the file input plug-in to read data from the log file, and you can also use the grok filter plug-in to parse the fields in the log file and add them to the event, and finally send the parsed data to the specified index of ES.

[0096] 4. Data consistency and update:

[0097] For data sources of different types and frequencies, regular incremental updates or real-time data synchronization are used to ensure that the data in ES is consistent with the data in the data source.

[0098] Data consistency: Use the Logstash input plugin to get data from different data sources, and then use the output plugin to send the data to the target storage ES. Among them, you can use the document_id parameter to specify the unique identifier of each data in the target storage ES to facilitate subsequent update and deletion operations. If the document_id parameter is not specified, Logstash will automatically generate a random identifier, which may cause data duplication or overwriting. Therefore, in order to ensure data consistency, it is recommended to generate the value of the document_id parameter based on one or more fields in the data source (such as a primary key or unique index).

[0099] Incremental data update: By using the schedule parameter to specify the frequency of regular incremental updates, such as every hour, every day, etc., Logstash will periodically obtain the latest data from the data source and determine whether to update or insert the data in the target storage ES based on the document_id parameter. To avoid obtaining the full amount of data, Logstash can use the sql_last_value parameter to record the timestamp or id of the last update and use it in the query statement to filter out the newly added or modified data.

[0100] Real-time data synchronization: Use the logstash-input-beats plugin to receive real-time data streams from Filebeat or other Beats and send them to the target storage ES. In this case, Logstash can use the metadata information (such as offset, inode, etc.) provided by Beats to generate the value of the document_id parameter, and use this value to determine whether the data in the target storage ES needs to be updated or inserted.

[0101] 5. Query and analysis:

[0102] At this point, multi-source data has been integrated into the target storage ES. The query and analysis interface provided by ES's powerful search capabilities is used to retrieve, aggregate and analyze the integrated multi-source data to obtain the required data results. Through a custom application, the data in the target storage ES is published as a unified external entrance and an interface document is generated. Data demand personnel can quickly obtain the required data and analyze the data by specifying indexes and query field conditions through the interface.

[0103] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application further provides a multi-source data fusion device.

[0105] like Figure 3 As shown, the multi-source data fusion device includes:

[0106] The identification and connection module 11 is configured to identify and connect data sources of different types and formats according to user input or configuration files;

[0107] The conversion module 12 is configured to perform corresponding data format conversion and mapping operations according to the data source, and convert the original data into a JSON document;

[0108] The data import module 13 is configured to import the JSON document into the search data analysis engine and create a corresponding index or update an existing index;

[0109] The data acquisition module 14 retrieves, aggregates and analyzes multi-source data through a search data analysis engine to obtain target data results.

[0110] For the convenience of description, the above devices are described in terms of functions and are divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0111] The device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0112] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0113] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0114] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0115] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0116] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0117] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0118] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0119] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0120] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0121] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0122] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0123] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0124] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0125] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0126] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0127] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A multi-source data fusion method, It is characterized in that include: Identify and connect to data sources of different types and formats based on user input or configuration files; Perform corresponding data format conversion and mapping operations according to the data source to convert the original data into a JSON document; Import the JSON document into the search data analysis engine and create a corresponding index or update an existing index; The search data analysis engine is used to retrieve, aggregate and analyze multi-source data to obtain target data results.

2. The method according to claim 1, Features: The data source includes at least a relational database, a non-relational database, an API interface and a log file.

3. The method according to claim 2, It is characterized in that The performing corresponding data format conversion and mapping operations according to the data source to convert the original data into a JSON document includes: The relational database includes MySQL, and the MySQL is configured to store a user information table and an order information table, convert each row of records in the information table into a JSON document, and add metadata information; The non-relational database includes MongoDB, and the MongoDB is configured to store a product information set, treat each document in the information set as a JSON document, and add metadata information; Perform corresponding data request and response operations according to the input and output of the API interface, convert the response result into a JSON document, and add metadata information; Convert each line of the log file into a JSON document and add metadata information.

4. The method according to claim 3, Features: The relational database also includes SQLServer, which creates a separate index for each standard data asset library. The index is subdivided into multiple independent types according to different basic information business tables. For each type, a common public field mapping is abstracted, standard data types and attributes are specified, each piece of data is converted into a mapped JSON document, and basic metadata information is supplemented.

5. The method according to claim 1, It is characterized in that Also includes: For data sources of different types and frequencies, the data in the search data analysis engine is made consistent with the data in the data source through regular incremental updates or real-time data synchronization.

6. The method according to claim 1, It is characterized in that The searching, aggregating and analyzing operations on multi-source data by the search data analysis engine to obtain target data results include: The data in the search data analysis engine is published as a unified external entrance through a customized application and an interface document is generated. The data demander obtains the target data results by passing the specified index and query field conditions through the interface, and analyzes and processes the data results.

7. A multi-source data fusion device, It is characterized in that include: An identification and connection module is configured to identify and connect data sources of different types and formats according to user input or configuration files; A conversion module is configured to perform corresponding data format conversion and mapping operations according to the data source to convert the original data into a JSON document; A data import module is configured to import the JSON document into the search data analysis engine and create a corresponding index or update an existing index; The data acquisition module retrieves, aggregates and analyzes multi-source data through the search data analysis engine to obtain target data results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, It is characterized in that in, The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.

Images