Multi-source data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-source data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-source data, carrying out the multi-source entity relationship merging processing of the multi-source data through a data stream execution engine, obtaining a merged data stream, carrying out the primary key integration processing of the merged data stream, and carrying out the secondary key integration processing of the merged data stream. And storing the integrated primary key data into a graph database, and performing graph data traceability query processing on the graph database to obtain a data query result. The data processing efficiency can be improved, and the method can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-source data processing method, device, electronic device and storage medium. Background Art

[0002] In the current data-driven business environment, enterprises need to process and integrate large and complex data from multiple data sources. These data sources are not only highly heterogeneous, but also have significant differences in format, structure and semantics. In the related art, there is a database management system that stores complex entity relationship data in a traditional relational database. However, this method is difficult to effectively express and manage complex many-to-many relationships between entities, and it is also difficult to meet the needs of large-scale data processing and fast query, which affects the efficiency and effectiveness of data integration and management.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a multi-source data processing method, device, electronic device and storage medium, which can improve data processing efficiency.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application provides a multi-source data processing method, the method comprising:

[0006] Obtain data from multiple sources;

[0007] Perform multi-source entity relationship merging processing on the multi-source data through a data stream execution engine to obtain a merged data stream;

[0008] Performing primary key integration processing on the merged data stream, and storing the integrated primary key data in a graph database;

[0009] Perform graph data source tracing query processing on the graph database to obtain data query results.

[0010] In some embodiments, the step of performing multi-source entity relationship merging processing on the multi-source data by a data stream execution engine to obtain a merged data stream includes the following steps:

[0011] Performing data reading processing on the multi-source data by the data stream execution engine to obtain read data;

[0012] Performing grouping processing on the read data according to the partition operator to obtain grouped data;

[0013] Aggregate the grouped data according to the specified aggregation operator to obtain aggregated data;

[0014] The aggregated data is merged to obtain the merged data stream.

[0015] In some embodiments, the merging of the aggregated data to obtain the merged data stream includes the following steps:

[0016] Obtaining policy configuration data, and saving the policy configuration data to state information;

[0017] Loading the merge strategy configuration file through the initialization function, and initializing the state information to obtain the data merge strategy;

[0018] The aggregated data is subjected to field comparison processing according to the data merging strategy to obtain the merged data stream.

[0019] In some embodiments, performing primary key integration processing on the merged data stream and storing the integrated primary key data in a graph database includes the following steps:

[0020] Acquire state information, and add the primary key value of the state information to the primary key storage field of the merged data stream to obtain the integrated primary key data;

[0021] Obtain database parameter information, and store the primary key data in the graph database according to the database parameter information.

[0022] In some embodiments, the acquiring of database parameter information and storing the primary key data in the graph database according to the database parameter information comprises the following steps:

[0023] Initializing the database parameter information to obtain initialization data;

[0024] Performing configuration processing on the entity relationship writing of the graph database according to the initialization data to obtain configuration writing parameters;

[0025] The primary key data is stored in the graph database according to the output operator in combination with the configuration write parameters.

[0026] In some embodiments, performing graph data source tracing query processing on the graph database to obtain data query results includes the following steps:

[0027] Performing entity association relationship query processing on the graph database to obtain graph data;

[0028] Primary key data query processing is performed according to the graph data to obtain the data query result.

[0029] In some embodiments, performing primary key data query processing according to the graph data to obtain the data query result includes the following steps:

[0030] Performing primary key data extraction processing on the graph data to obtain a query primary key list;

[0031] Data query processing is performed on the relational database according to the query primary key list to obtain the data query result.

[0032] To achieve the above objective, another aspect of an embodiment of the present application provides a multi-source data processing device, the device comprising:

[0033] The first module is used to obtain multi-source data;

[0034] The second module is used to perform multi-source entity relationship merging processing on the multi-source data through a data flow execution engine to obtain a merged data flow;

[0035] The third module is used to perform primary key integration processing on the merged data stream and store the integrated primary key data into a graph database;

[0036] The fourth module is used to perform graph data traceability query processing on the graph database to obtain data query results.

[0037] In some embodiments, the second module is used to perform multi-source entity relationship merging processing on the multi-source data through a data flow execution engine to obtain a merged data flow, including:

[0038] A first unit is used to perform data reading processing on the multi-source data through the data stream execution engine to obtain read data;

[0039] The second unit is used to perform grouping processing on the read data according to the partition operator to obtain grouped data;

[0040] A third unit is used to perform aggregation processing on the grouped data according to a specified aggregation operator to obtain aggregated data;

[0041] The fourth unit is used to merge the aggregated data to obtain the merged data stream.

[0042] In some embodiments, the fourth unit is used to merge the aggregated data to obtain the merged data stream, including:

[0043] A first subunit is used to obtain policy configuration data and save the policy configuration data to state information;

[0044] The second subunit is used to load the merge strategy configuration file through the initialization function, and initialize the state information to obtain the data merge strategy;

[0045] The third subunit is used to perform field comparison processing on the aggregated data according to the data merging strategy to obtain the merged data stream.

[0046] In some embodiments, the third module is used to perform primary key integration processing on the merged data stream and store the integrated primary key data in the graph database, including:

[0047] A fifth unit is used to obtain state information, and add the primary key value of the state information to the primary key storage field of the merged data stream to obtain the integrated primary key data;

[0048] The sixth unit is used to obtain database parameter information, and store the primary key data in the graph database according to the database parameter information.

[0049] In some embodiments, the sixth unit, used to obtain database parameter information and store the primary key data in the graph database according to the database parameter information, includes:

[0050] The fourth subunit is used to initialize the database parameter information to obtain initialization data;

[0051] A fifth subunit is used to perform entity relationship writing execution configuration processing on the graph database according to the initialization data to obtain configuration writing parameters;

[0052] The sixth subunit is used to store the primary key data into the graph database according to the output operator in combination with the configuration write parameters.

[0053] In some embodiments, the fourth module is used to perform graph data source tracing query processing on the graph database to obtain data query results, including:

[0054] The seventh unit is used to perform entity association relationship query processing on the graph database to obtain graph data;

[0055] The eighth unit is used to perform primary key data query processing according to the graph data to obtain the data query result.

[0056] In some embodiments, the eighth unit is used to perform primary key data query processing according to the graph data to obtain the data query result, including:

[0057] A seventh subunit is used to extract primary key data from the graph data to obtain a query primary key list;

[0058] The eighth subunit is used to perform data query processing on the relational database according to the query primary key list to obtain the data query result.

[0059] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0060] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0061] The embodiments of the present application include at least the following beneficial effects: The present application provides a multi-source data processing method, device, electronic device and storage medium. The scheme obtains multi-source data, and performs multi-source entity relationship merging processing on the multi-source data through a data flow execution engine to obtain a merged data flow. By quickly merging multi-source data, the efficiency of data processing is improved, and the possibility of data redundancy and information loss is significantly reduced. In addition, the scheme performs primary key integration processing on the merged data flow, and stores the integrated primary key data in a graph database, performs graph data traceability query processing on the graph database, and obtains data query results. The primary key data can be stored in the graph database to form a complete data traceability path, thereby improving the consistency and accuracy of the data, and enhancing the availability and transparency of the data, thereby improving the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of a multi-source data processing method provided by an embodiment of the present application;

[0063] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0064] Figure 3 yes Figure 2 Flow chart of step S204 in FIG.

[0065] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0066] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0067] Figure 6 It is a flow chart of a merging logic provided by an embodiment of the present application;

[0068] Figure 7This is a flow chart of a primary key integration provided by an embodiment of the present application;

[0069] Figure 8 This is a flow chart of data tracing provided by an embodiment of the present application;

[0070] Fig. 9 is a structural schematic diagram of a multi-source data processing device provided in an embodiment of the present application;

[0071] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0073] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0074] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0076] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0077] 1) Flink, also known as Apache Flink, is an open source stream processing framework for processing unbounded and bounded data streams with high throughput and low latency.

[0078] 2) Doris, also known as Apache Doris, is a high-performance, real-time analytical database based on the MPP architecture. The MPP (Massively Parallel Processing) architecture is a distributed data processing technology that can improve data processing performance by distributing the workload across multiple nodes.

[0079] 3) Nebula database is a proprietary or domain-specific graph database or distributed storage system that focuses on storing and querying complex graph structured data and supports primary key traceability.

[0080] 4) Multi-source entity relationship merging: extracting entities and their relationship information from multiple data sources, merging, deduplicating, resolving conflicts, etc. through algorithms or rules to form a unified entity relationship diagram.

[0081] 5) Primary key traceability storage method, a data storage and query technology that tracks and traces data changes by maintaining the primary key of the data and its change history.

[0082] In the digital environment, enterprises and organizations are facing increasingly complex data management challenges. These challenges mainly come from the following aspects:

[0083] First, the existence of multi-source heterogeneous data leads to the complexity of data integration and analysis. Multi-source heterogeneous data has the characteristics of diversity of source data and differentiated data structure. For example, enterprises obtain data from multiple channels, which may come from different business systems, sensor devices, social media platforms, etc. Each data source has its own unique data format and attribute definition, which brings challenges to extracting valuable entity relationships from multiple data sources.

[0084] Secondly, the data merging methods in related technologies and the limitations of relational databases limit the effective integration of data. Traditional methods often fail to merge data efficiently when facing multi-source data, resulting in data redundancy and consistency problems. The inconsistency of attributes of different source data causes multiple records of the same entity to contain a large amount of duplicate information, increasing storage costs and maintenance complexity. In addition, information loss or inaccuracy may occur during data merging, further affecting the quality and use value of the data. At the same time, most companies usually store complex entity relationship data in traditional relational databases, taking Doris as an example. Although relational databases have advantages in transaction processing and data storage, they are not ideal for managing complex entity relationship data. Conventional databases usually store data in the form of tables, which makes it difficult to effectively express and manage complex many-to-many relationships between entities, and it is also difficult to meet the needs of large-scale data processing and fast query. These limitations jointly restrict the efficiency and effectiveness of enterprises in data integration and management.

[0085] Finally, there is the problem of insufficient traceability of data sources. With the diversification of data usage scenarios, enterprises need to ensure the traceability of data sources. However, traditional database management systems often fail to provide sufficiently fine-grained and flexible data traceability mechanisms, especially for complex situations involving multi-source data integration. This makes it difficult for enterprises to accurately trace the source and change history of data, affecting the credibility and compliance of data.

[0086] In view of this, a multi-source data processing method, device, electronic device and storage medium are provided in the embodiments of the present application. The scheme obtains multi-source data, and performs multi-source entity relationship merging processing on the multi-source data through a data flow execution engine to obtain a merged data flow. The scheme can improve the efficiency of data processing by quickly merging multi-source data, and significantly reduce the possibility of data redundancy and information loss. In addition, the scheme performs primary key integration processing on the merged data flow, and stores the integrated primary key data in a graph database, and performs graph data traceability query processing on the graph database to obtain data query results. The scheme can store the primary key data in the graph database to form a complete data traceability path, thereby improving the consistency and accuracy of the data, and also enhancing the availability and transparency of the data, and improving the efficiency of data processing.

[0087] A multi-source data processing method provided in an embodiment of the present application relates to the field of data processing technology. A multi-source data processing method provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a multi-source data processing method, etc., but is not limited to the above forms.

[0088] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0089] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0090] Figure 1 is an optional flowchart of a multi-source data processing method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0091] Step S101, acquiring multi-source data;

[0092] Step S102, performing multi-source entity relationship merging processing on the multi-source data through a data flow execution engine to obtain a merged data flow;

[0093] Step S103, performing primary key integration processing on the merged data stream, and storing the integrated primary key data in a graph database;

[0094] Step S104, performing graph data source tracing query processing on the graph database to obtain data query results.

[0095] Steps S101 to S104 shown in the embodiment of the present application are obtained by acquiring multi-source data, which is data obtained from multiple channels and can come from different business systems, sensor devices, social media platforms, etc. The multi-source data is merged by the data stream execution engine to obtain a merged data stream. The embodiment of the present application adopts Flink as the data stream execution engine, and the consistency and accuracy of data integration are improved by quickly merging the multi-source data. Through the efficient data processing mechanism of the data stream execution engine, different source data can be quickly merged, the efficiency of data processing is improved, and the possibility of data redundancy and information loss is reduced. Then the merged data stream is processed by primary key integration, and the integrated primary key data is stored in the graph database, which can record the key information of each data processing step in detail, including the source of the data, processing time and processing results. The embodiment of the present application forms a complete data traceability path by storing these key information in the form of primary keys in the graph database. The embodiment of the present application not only improves the consistency and accuracy of the data, but also enhances the availability and transparency of the data. Finally, the graph database is processed by graph data traceability query to obtain data query results, which can quickly query the target data and improve the efficiency of data processing.

[0096] In step S101 of some embodiments, multi-source data can be obtained from different business systems, sensor devices, social media platforms, etc., and then stored in a relational database. The relational database can adopt a Doris database. The multi-source data can be stored in the relational database to facilitate subsequent query of the multi-source data according to the primary key data.

[0097] See also Figure 2 In step S102 of some embodiments, the multi-source entity relationship merging process is performed on the multi-source data by the data flow execution engine to obtain a merged data flow, including the following steps:

[0098] Step S201, performing data reading processing on the multi-source data by the data flow execution engine to obtain read data;

[0099] Step S202, grouping the read data according to the partition operator to obtain grouped data;

[0100] Step S203, performing aggregation processing on the grouped data according to a specified aggregation operator to obtain aggregated data;

[0101] Step S204, merging the aggregated data to obtain the merged data stream.

[0102] In the embodiment of the present application, the data stream execution engine performs multi-source entity relationship merging processing on multi-source data to obtain a merged data stream, wherein the data stream execution engine adopts the Flik stream processing framework. In Flink, it is possible to select rich functions (RichFunction) different from conventional functions to access runtime context information during data processing, such as the parallelism of tasks, task names, and state management interfaces. This information is essential for implementing complex data processing logic, monitoring, and debugging stream processing jobs. In addition, the rich function also provides a method for life cycle management, including performing necessary resource allocation and configuration and connection initialization of third-party components in the initialization phase (open method) before the task starts, and releasing resources in the cleanup phase (close method) after the task ends, that is, closing the connection of third-party components, etc., which helps to optimize the performance and resource utilization of the application. Specifically, by performing data reading processing on multi-source data, taking the Doris database as an example, by setting the Doris connection information, the Doris connection constructor, and the Doris table reading settings, the source data (Doris Source) is converted into stream data (DataStream) to obtain read data. It is conceivable that the embodiment of the present application can also periodically pull data by obtaining the change log of the Doris database or using polling queries, or read the data in the Doris database by using middleware, such as using Apache Kafka message queue as middleware, sending data changes to Kafka through the Doris database, and then subscribing to the Kafka topic through Flink to process the data in real time. Then, the read data is grouped according to the partition operator to obtain grouped data, wherein the partition operator is a keyBy operator, which is used to group the data stream according to the specified key (key), which is conducive to subsequent aggregation operations, and the grouping key is the entity and association relationship type, which can make the same entity and association relationship be divided into the same parallelism. Then, the grouped data is aggregated according to the reduction aggregation operator to obtain aggregated data, wherein the reduction aggregation operator is a reduce operator, which is used to perform aggregation operations on the elements in the data stream, by combining the elements in the data stream in pairs, and applying a specified function to merge the two elements until only one element is left in the data stream. Finally, the aggregated data is merged to obtain a merged data stream. The merge process adopts a merge strategy, and the merge logic is executed by loading the merge strategy to obtain a merged data stream. The embodiment of the present application performs multi-source entity relationship merge processing on multi-source data through a data stream execution engine, which can quickly merge multi-source data, improve the efficiency of data processing, and reduce the possibility of data redundancy and information loss.

[0103] See also Figure 3In step S204 of some embodiments, the merging of the aggregated data to obtain the merged data stream includes the following steps:

[0104] Step S301, obtaining policy configuration data, and saving the policy configuration data to state information;

[0105] Step S302, loading a merging strategy configuration file through an initialization function, and initializing the state information to obtain a data merging strategy;

[0106] Step S303: Perform field comparison processing on the aggregated data according to the data merging strategy to obtain the merged data stream.

[0107] In an embodiment of the present application, policy configuration data is obtained, and the policy configuration data is saved to state information, wherein an implementation class of a rich function interface is created by Flink, and a state attribute of a ValueState type is provided in the implementation class as state information, and ValueState is a state for storing a single value associated with a specific key (Key). The state information is used to store policy configuration data, and the policy configuration data can be a combination of parameters, rules or algorithms, and the parameters can be the memory allocation of the system, the number of processor cores, the timeout time, etc., or can be a parameter related to a specific business logic, and the policy configuration data can guide the system how to automatically perform corresponding operations or selections according to different conditions, environments and goals. Then the merge strategy configuration file is loaded by the initialization function, and the open method is rewritten to initialize the state information according to the merge strategy configuration file, so that the merge strategies of different entity relationship types are different. Finally, the aggregated data is subjected to field comparison processing according to the data merge strategy to obtain a merged data stream, wherein different entity data types use different merge strategies, and when the data stream is merged, different entity types are merged according to the strategy of the corresponding entity type in the merge strategy. Specifically, the merge strategy is to compare multiple fields between data, and the comparison method includes whether there is a value, the legality of the value boundary, the value uniqueness, the value size, and the joint uniqueness of multiple values, etc. For example, the value uniqueness of multiple data of the same entity relationship is judged, and the data judged to have value uniqueness is merged. The embodiment of the present application adopts different merge strategies to merge different entity data types, which can clean and deduplicate the data during the merge process, eliminate errors and redundant information, help improve the accuracy and consistency of the data, reduce errors caused by duplicate, missing or inaccurate data, and improve data processing efficiency.

[0108] See also Figure 4In step S103 of some embodiments, the step of performing primary key integration processing on the merged data stream and storing the integrated primary key data in a graph database includes the following steps:

[0109] Step S401, acquiring state information, and adding the primary key value of the state information to the primary key storage field of the merged data stream to obtain the integrated primary key data;

[0110] Step S402, obtaining database parameter information, and storing the primary key data in the graph database according to the database parameter information.

[0111] In an embodiment of the present application, an implementation class is created through Flink, and there is a state attribute of MapState type in the implementation class as state information, wherein MapState is a state backend for storing key-value pair mappings, which can store a value associated with each key (Key), and this value itself is a mutable mapping (Map), that is, multiple key-value pairs can be stored in this mapping. In the open method, the embodiment of the present application initializes the state information, wherein Key is a value of different entity types, and Value is the primary key of all detail data corresponding to different entity relationships, and the primary key value of this state information is added to the primary key storage field (detail_ids) of the merged data stream data to obtain the integrated primary key data. Then, by obtaining the database parameter information, the primary key data is stored in the graph database. The embodiment of the present application forms a complete data traceability path by storing the primary key information of each data processing step in the graph database, which not only improves the consistency and accuracy of the data, but also enhances the availability and transparency of the data and improves the data processing efficiency.

[0112] See also Figure 5 In step S402 of some embodiments, the acquiring of database parameter information and the storing of the primary key data in the graph database according to the database parameter information include the following steps:

[0113] Step S501, initializing the database parameter information to obtain initialization data;

[0114] Step S502, performing configuration processing on the entity relationship writing of the graph database according to the initialization data to obtain configuration writing parameters;

[0115] Step S503: store the primary key data into the graph database according to the output operator in combination with the configuration write parameters.

[0116] In an embodiment of the present application, the primary key data is stored in a graph database according to the database parameter information, wherein the graph database adopts the Nebula Graph graph database. Specifically, the graph database parameter information is initialized, and the configuration of the Neblua write client is created, including the connection information such as the user password of Nebula's Graph service and Meta service, wherein the Graph service is the service responsible for processing calculation requests in Nebula Graph, and is the calculation layer provided by the kernel, which directly receives external operations such as graph library login, graph query requests, and cluster management; the Meta service is the service responsible for managing metadata in Nebula Graph, which is responsible for storing and providing metadata of graph data, such as graph space, schema definition, user authentication and authorization information, etc. Then, according to the initialization data, the graph database is subjected to entity relationship write execution configuration processing to obtain the configuration write parameters, and the edge point information is configured by creating an entity relationship write execution, which includes the graph space, edge name, location index of the data set where the edge start and end points are located, field location index set, write mode, and write batch size, etc. Finally, the configuration write parameters are assembled into the NebulaSinkFunction function, which is used to write the Flink data stream into the Nebula database. Specifically, the data stream is written into the Nebula database through the Flink output operator. The output operator is the addSink operator, which is used to specify the end point of the data stream processing pipeline, that is, the final destination of the data flow, so that Flink can output the processing results to the graph database in real time for other systems or users to perform further processing or analysis. The embodiment of the present application ensures the traceability and integrity of the data by storing the primary key data in the graph database. The graph database can record the change history of the data in detail, support the tracking of data sources and changes, and form a clear data lineage relationship. This not only improves the consistency and accuracy of the data, but also enhances the transparency and reliability of data management.

[0117] In some embodiments, performing graph data source tracing query processing on the graph database to obtain data query results includes the following steps:

[0118] Performing entity association relationship query processing on the graph database to obtain graph data;

[0119] Primary key data query processing is performed according to the graph data to obtain the data query result.

[0120] In an embodiment of the present application, graph data can be obtained by performing entity association relationship query processing on a graph database. The graph data is primary key data and contains entity data and the association relationship between entity data. Then, a primary key data query is performed based on the graph data, and a data query can be performed on multi-source data based on the primary key data to obtain a data query result, which includes the target data obtained by the query and the query result being empty. Taking the Nebula graph database as an example, Nebula's nGQL syntax is used to query Nebula, where nGQL (Nebula Graph Query Language) is a declarative graph query language used by Nebula Graph, allowing users to perform complex traversal queries in the graph database by defining the connection relationship between points and edges. The embodiment of the present application performs data query through a graph database, which can improve data traceability and query efficiency.

[0121] In some embodiments, performing primary key data query processing according to the graph data to obtain the data query result includes the following steps:

[0122] Performing primary key data extraction processing on the graph data to obtain a query primary key list;

[0123] Data query processing is performed on the relational database according to the query primary key list to obtain the data query result.

[0124] In an embodiment of the present application, primary key data is extracted from graph data, and the value in the primary key storage field (detail_ids) is extracted to obtain a query primary key list, thereby performing a multi-source data query based on the obtained detail primary key list. The embodiment of the present application can save all multi-source data to a relational database, and then query the relational database based on the primary key list, use standard SQL syntax to query, and return all queried data. By storing primary key data in a graph database, the embodiment of the present application can enhance the transparency and reliability of data management, help enterprises better understand and manage the life cycle of data, and improve data query efficiency when querying graph data.

[0125] The following is a detailed description of the embodiments of the present invention with reference to specific application examples:

[0126] The embodiments of the present application can be applied to the field of data processing technology, and specifically to multi-source data processing scenarios, such as smart manufacturing, smart transportation, energy management, environmental detection management and other application scenarios. For example, in smart manufacturing application scenarios, it is necessary to perform intelligent and automated control of products. By collecting various data, such as the operating status of production equipment, quality parameters of raw materials, etc., the intelligent and automated control of products can be achieved to improve production efficiency and product quality. At this time, it is necessary to process multi-source data. The embodiments of the present application quickly merge and process multi-source data, and perform graph database traceability storage on primary keys. Through efficient data merging and processing and data traceability management, the quality and business value of data are improved. For details, please refer to Figure 6 The embodiment of the present application adopts the Flink stream data processing framework, uses the keyBy operator to group the data stream, enters the merge logic and uses the reduce operator to aggregate the grouped data. In the rich function (RichReduceFunction), the ValueState state attribute is used to store the policy configuration data, which can be parameters, rules, algorithms, etc. The merge strategy configuration file is loaded and the state attribute is initialized in the initialization function (open). Different merge strategies are used for different entity relationship types. Multiple data of the same entity relationship are merged according to the merge strategy. The merge strategy can be whether there is a value, the legality of the value boundary, the uniqueness of the value, the joint uniqueness of multiple values, the size of the value, etc. The merged data stream is obtained by merging the hit data through the strategy. Please refer to Figure 7 In the embodiment of the present application, data is read from a relational database through a conversion function (Flink Doris Source), the data in the Doris database is converted into stream data (Data Stream), and then the merge logic is executed, and grouping processing is performed through a grouping function (KeyByStream), specifically using the keyBy operator for grouping, where the grouping key is the entity and associated relationship value, and then multiple data of the same entity relationship value are merged to obtain a merged data stream. The merged data stream is integrated with the primary key of the detailed data, and the value in the state attribute is added to the primary key storage field (detail_ids) of the final data of the merged data stream to obtain the integrated primary key data and write the data into the Nebula graph database. Please refer to Figure 8, the embodiment of the present application adopts Nebula as a graph database, which stores graph data, including entity data, relationship data and associated data. By querying the entity association relationship of the graph database, the detailed primary key data corresponding to the graph data is read, and the detailed data is obtained based on the primary key query. The embodiment of the present application can provide an innovative data management and analysis solution for enterprises. By improving the efficiency and accuracy of data merging, enhancing the consistency and availability of data, and realizing the traceability of data, it is expected to empower enterprises in digital transformation and data-driven decision-making, and promote the continuous innovation and development of business.

[0127] See also Fig. 9 The embodiment of the present application further provides a multi-source data processing device, which can implement the multi-source data processing method described above, and the device includes:

[0128] The first module 901 is used to obtain multi-source data;

[0129] The second module 902 is used to perform multi-source entity relationship merging processing on the multi-source data through a data flow execution engine to obtain a merged data flow;

[0130] The third module 903 is used to perform primary key integration processing on the merged data stream and store the integrated primary key data into a graph database;

[0131] The fourth module 904 is used to perform graph data source tracing query processing on the graph database to obtain data query results.

[0132] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above multi-source data processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0134] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0136] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0137] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the multi-source data processing method of the embodiment of this application;

[0138] Input / output interface 1003, used to implement information input and output;

[0139] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0140] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0141] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0142] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the multi-source data processing method is implemented.

[0143] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The embodiments of the present application provide a multi-source data processing method, device, electronic device and storage medium. The solution obtains multi-source data, merges the multi-source entity relationships of the multi-source data through a data flow execution engine, and obtains a merged data flow. The solution can improve the efficiency of data processing by quickly merging multi-source data, and significantly reduce the possibility of data redundancy and information loss. In addition, the solution performs primary key integration processing on the merged data flow, stores the integrated primary key data in a graph database, performs graph data traceability query processing on the graph database, obtains data query results, and can store the primary key data in the graph database to form a complete data traceability path, thereby improving the consistency and accuracy of the data, enhancing the availability and transparency of the data, and improving the efficiency of data processing.

[0146] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0148] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0150] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0156] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multi-source data processing method, characterized in that: The method comprises the following steps: Obtain data from multiple sources; Perform multi-source entity relationship merging processing on the multi-source data through a data stream execution engine to obtain a merged data stream; Performing primary key integration processing on the merged data stream, and storing the integrated primary key data in a graph database; Perform graph data source tracing query processing on the graph database to obtain data query results.

2. The method according to claim 1, characterized in that The method of performing multi-source entity relationship merging processing on the multi-source data by a data stream execution engine to obtain a merged data stream includes the following steps: Performing data reading processing on the multi-source data by the data stream execution engine to obtain read data; Performing grouping processing on the read data according to the partition operator to obtain grouped data; Aggregate the grouped data according to the specified aggregation operator to obtain aggregated data; The aggregated data is merged to obtain the merged data stream.

3. The method according to claim 2, characterized in that The merging of the aggregated data to obtain the merged data stream includes the following steps: Obtaining policy configuration data, and saving the policy configuration data to state information; Loading the merge strategy configuration file through the initialization function, and initializing the state information to obtain the data merge strategy; The aggregated data is subjected to field comparison processing according to the data merging strategy to obtain the merged data stream.

4. The method according to claim 1, characterized in that: The step of performing primary key integration processing on the merged data stream and storing the integrated primary key data in a graph database includes the following steps: Acquire state information, and add the primary key value of the state information to the primary key storage field of the merged data stream to obtain the integrated primary key data; Obtain database parameter information, and store the primary key data in the graph database according to the database parameter information.

5. The method according to claim 4, characterized in that The acquiring of database parameter information and storing the primary key data in the graph database according to the database parameter information comprises the following steps: Initializing the database parameter information to obtain initialization data; Performing configuration processing on the entity relationship writing of the graph database according to the initialization data to obtain configuration writing parameters; The primary key data is stored in the graph database according to the output operator in combination with the configuration write parameters.

6. The method according to claim 1, characterized in that The step of performing graph data source tracing query processing on the graph database to obtain data query results includes the following steps: Performing entity association relationship query processing on the graph database to obtain graph data; Primary key data query processing is performed according to the graph data to obtain the data query result.

7. The method according to claim 6, characterized in that The step of performing primary key data query processing according to the graph data to obtain the data query result includes the following steps: Performing primary key data extraction processing on the graph data to obtain a query primary key list; Data query processing is performed on the relational database according to the query primary key list to obtain the data query result.

8. A multi-source data processing device, characterized in that: The device comprises: The first module is used to obtain multi-source data; The second module is used to perform multi-source entity relationship merging processing on the multi-source data through a data flow execution engine to obtain a merged data flow; The third module is used to perform primary key integration processing on the merged data stream and store the integrated primary key data into a graph database; The fourth module is used to perform graph data traceability query processing on the graph database to obtain data query results.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images