Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of big data, the method comprises the following steps: determining initialization functions for a first database and a second database; the first database and the second database are initialized according to the initialization function, and connection between the distributed stream processing framework and the first database and connection between the distributed stream processing framework and the second database are established; responding to the received data stream, wherein the data stream comprises at least one piece of service data; extracting at least one key field used for identifying the service data from the service data, and determining a data identifier corresponding to the service data according to the at least one key field; and writing the business data into the first database according to the data identifier, and writing the business data into the second database under the condition that the business data is successfully written into the first database, so that the consistency and integrity of data among different databases are ensured.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, a data processing device, an electronic device and a computer-readable storage medium. Background Art

[0002] With the development of big data technology, real-time processing and storage of massive data has become an important technical challenge. Among them, a single database cannot meet the diverse data needs in different scenarios, so multiple databases coexist. However, when multiple databases coexist, it is difficult to ensure the consistency and integrity of data between different databases in the case of high data concurrency, which affects the application of data. Summary of the invention

[0003] The embodiments of the present invention provide a data processing method, device, electronic device and computer-readable storage medium to solve or partially solve the problem of poor consistency and integrity of data among multiple databases.

[0004] The embodiment of the present invention discloses a data processing method, involving a first database and a second database, wherein the first database is at least used for processing the storage, management and query of a graph data structure, and the second database is at least used for full-text retrieval, structured retrieval, log analysis and data indexing in a big data environment; wherein the processing method is applied to a distributed stream processing framework, including:

[0005] Determine initialization functions for the first database and the second database;

[0006] Initialize the first database and the second database according to the initialization function, and establish connections between the distributed stream processing framework and the first database and the second database;

[0007] In response to receiving a data stream, the data stream includes at least one piece of service data;

[0008] Extracting at least one key field for identifying the business data from the business data, and determining a data identifier corresponding to the business data according to the at least one key field;

[0009] The business data is written into the first database according to the data identifier, and if the business data is successfully written into the first database, the business data is written into the second database.

[0010] In some feasible implementations, the first database includes a connection pool, the initialization function includes at least a connection pool initialization function corresponding to the connection pool, and the first database and the second database are initialized according to the initialization function, and the connection between the distributed stream processing framework and the first database and the second database is established, including:

[0011] Initialize the connection pool through the connection pool initialization function, set the connection pool to an available state, and establish a first connection between the first database and the distributed stream processing framework;

[0012] A database instance for the second database is obtained through the initialization function, and the second database is initialized according to the database instance, so as to establish a second connection between the second database and the distributed stream processing framework.

[0013] In some feasible implementations, determining the data identifier corresponding to the business data according to at least one of the key fields includes:

[0014] Concatenate at least one of the key fields into a character string corresponding to the business data;

[0015] The character string is converted into a data identifier that matches the business data.

[0016] In some feasible implementations, the key field includes at least one of a user identifier, a timestamp, and an event type, and the step of concatenating at least one of the key fields into a character string corresponding to the business data includes:

[0017] The user identifier, the timestamp, and one of the event types are concatenated into a character string corresponding to the business data.

[0018] In some feasible implementations, converting the character string into a data identifier that matches the business data includes:

[0019] The character string is used as an input of a hash function, an operation is performed on the character string to obtain a hash value matching the business data, and the hash value is used as a data identifier of the business data.

[0020] In some feasible implementations, writing the business data into the first database according to the data identifier, and if the business data is successfully written into the first database, writing the business data into the second database includes:

[0021] Writing the business data into the first database according to the data identifier;

[0022] If it is detected that the business data fails to be written into the first database, the distributed stream processing framework is triggered to re-write the business data into the first database according to the data identifier;

[0023] If it is detected that the business data is successfully written into the first database, the business data is written into the second database according to the data identifier.

[0024] In some feasible implementations, writing the business data into the second database according to the data identifier includes:

[0025] If it is detected that the business data fails to be written into the second database, the distributed stream processing framework is triggered to re-write the business data into the second database according to the data identifier.

[0026] Some possible implementations also include:

[0027] In response to a data query instruction for the second database, querying the second database for target business data corresponding to the data query instruction;

[0028] If the target business data is found in the second database, receiving the target business data returned by the second database;

[0029] If the target business data is not found in the second database, searching the first database for the target business data corresponding to the data query instruction;

[0030] If the target service data is found in the first database, receiving the target service data returned by the first database;

[0031] If the target business data is not found in the first database, a failure prompt message for the data query instruction is output, where the failure prompt message is used to prompt that the relevant data is not found.

[0032] The embodiment of the present invention further discloses a data processing device, involving a first database and a second database, wherein the first database is at least used for processing the storage, management and query of a graph data structure, and the second database is at least used for full-text retrieval, structured retrieval, log analysis and data indexing in a big data environment; wherein the processing device is applied to a distributed stream processing framework, and comprises:

[0033] A function determination module, used to determine initialization functions for the first database and the second database;

[0034] An initialization module, configured to initialize the first database and the second database according to the initialization function, and establish a connection between the distributed stream processing framework and the first database and the second database;

[0035] A data receiving module, configured to respond to receiving a data stream, wherein the data stream includes at least one piece of service data;

[0036] an identification generating module, used to extract at least one key field for identifying the business data from the business data, and determine a data identification corresponding to the business data according to at least one key field;

[0037] A data writing module is used to write the business data into the first database according to the data identifier, and write the business data into the second database if the business data is successfully written into the first database.

[0038] In some feasible implementations, the first database includes a connection pool, the initialization function includes at least a connection pool initialization function corresponding to the connection pool, and the initialization module is specifically used to:

[0039] Initialize the connection pool through the connection pool initialization function, set the connection pool to an available state, and establish a first connection between the first database and the distributed stream processing framework;

[0040] A database instance for the second database is obtained through the initialization function, and the second database is initialized according to the database instance, so as to establish a second connection between the second database and the distributed stream processing framework.

[0041] In some feasible implementations, the identification generation module is specifically used to:

[0042] Concatenate at least one of the key fields into a character string corresponding to the business data;

[0043] The character string is converted into a data identifier that matches the business data.

[0044] In some feasible implementations, the key field includes at least one of a user identifier, a timestamp, and an event type, and the identifier generation module is specifically used to:

[0045] The user identifier, the timestamp, and one of the event types are concatenated into a character string corresponding to the business data.

[0046] In some feasible implementations, the identification generation module is specifically used to:

[0047] The character string is used as an input of a hash function, an operation is performed on the character string to obtain a hash value matching the business data, and the hash value is used as a data identifier of the business data.

[0048] In some feasible implementations, the data writing module is specifically used to:

[0049] Writing the business data into the first database according to the data identifier;

[0050] If it is detected that the business data fails to be written into the first database, the distributed stream processing framework is triggered to re-write the business data into the first database according to the data identifier;

[0051] If it is detected that the business data is successfully written into the first database, the business data is written into the second database according to the data identifier.

[0052] In some feasible implementations, the data writing module is specifically used to:

[0053] If it is detected that the business data fails to be written into the second database, the distributed stream processing framework is triggered to re-write the business data into the second database according to the data identifier.

[0054] Some possible implementations also include:

[0055] A first query module, configured to, in response to a data query instruction for the second database, query the second database for target business data corresponding to the data query instruction;

[0056] a first data returning module, configured to receive the target business data returned by the second database if the target business data is found in the second database;

[0057] A second query module, configured to query the first database for the target business data corresponding to the data query instruction if the target business data is not found in the second database;

[0058] A second data returning module, configured to receive the target business data returned by the first database if the target business data is found in the first database;

[0059] A prompt module is used to output failure prompt information for the data query instruction if the target business data is not found in the first database, and the failure prompt information is used to prompt that the relevant data is not found.

[0060] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0061] The memory is used to store computer programs;

[0062] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0063] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0064] The embodiments of the present invention include the following advantages:

[0065] In an embodiment of the present invention, it is applied to a distributed stream processing framework. In the process of processing business data, an initialization function for a first database and a second database may be determined first, and then the first database and the second database may be initialized according to the initialization function to establish a connection between the distributed stream processing framework and the first database and the second database. By initializing the first database and the second database in an initialization function, a corresponding connection is established to ensure the consistency of the connection. In the process of writing data, in response to receiving a data stream, the data stream includes at least one business data, and then at least one key field for identifying the business data is extracted from the business data, and a data identifier corresponding to the business data is determined according to at least one key field. Then, the business data is written to the first database according to the data identifier, and when the first database is successfully written, the business data is written to the second database. By generating a unique data identifier corresponding to the business data, the uniqueness of the data is ensured, and while reducing duplicate writing, the consistency and integrity of the data between the first database and the second database are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flowchart of a method for processing data provided in an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of the initialization process provided in an embodiment of the present invention;

[0068] Figure 3 is a schematic diagram of a data writing process provided in an embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of data query provided in an embodiment of the present invention;

[0070] Figure 5 is a structural block diagram of a data processing device provided in an embodiment of the present invention;

[0071] Figure 6 It is a block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] As an example, with the development of big data technology, real-time processing and storage of massive data has become an important technical challenge. A single database cannot meet the diverse data needs in different scenarios. Therefore, multiple databases coexist. For example, in the financial big data scenario, when full-text retrieval is performed after writing data to the graph library, as the amount of data increases, the full-text retrieval capability of the graph database decreases significantly, making it difficult to meet business needs. In order to improve the full-text retrieval capability of the graph database under large amounts of data and ensure data consistency, a method based on the coexistence of multiple databases can support fast search. However, when multiple databases coexist, there are problems with poor data consistency and integrity.

[0074] In this regard, in the present invention, it is applied to a distributed stream processing framework. In the process of processing business data, the initialization function for the first database and the second database can be first determined, and then the first database and the second database are initialized according to the initialization function, and a connection between the distributed stream processing framework and the first database and the second database is established. By initializing the first database and the second database in an initialization function, a corresponding connection is established, and the consistency of the connection is ensured. In the process of writing data, in response to receiving a data stream, the data stream includes at least one business data, and then at least one key field for identifying the business data is extracted from the business data, and a data identifier corresponding to the business data is determined according to at least one key field, and then the business data is written to the first database according to the data identifier. In the case of successful writing to the first database, the business data is written to the second database. By generating a unique data identifier corresponding to the business data, the uniqueness of the data is ensured, and while reducing repeated writing, the consistency and integrity of the data between the first database and the second database are guaranteed.

[0075] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, some technical features involved in the embodiments of the present invention are explained below:

[0076] Apache Flink: is an open source stream processing framework that can perform distributed data processing and analysis. It supports both batch processing and stream processing modes and can process real-time data streams and historical data.

[0077] Dual-write: is a data consistency strategy commonly used in distributed systems or database systems. It writes data to two independent storage systems at the same time to ensure high availability and consistency of data.

[0078] ES (Elasticsearch): is a distributed search and analysis engine used for full-text search, structured search, log analysis, and data indexing in a big data environment.

[0079] Nebula (Nebula Graph): is an open source distributed graph database designed for processing large-scale graphs with billions of vertices and edges. It is suitable for the storage, management and query of graph data structures and is commonly used in social networks, recommendation systems, knowledge graphs and other fields.

[0080] Idempotent Operation: refers to a property that an operation can be performed multiple times, but the result is the same as if it were performed only once. No matter how many times the operation is performed, the effect is the same.

[0081] Specifically, refer to Figure 1 , shows a flow chart of the steps of a data processing method provided in an embodiment of the present invention, involving a first database and a second database, wherein the first database is at least used for storage, management and query of processing graph data structures, and the second database is at least used for full-text retrieval, structured retrieval, log analysis and data indexing in a big data environment; wherein the processing method is applied to a distributed stream processing framework, and may specifically include the following steps:

[0082] Step 101, determining initialization functions for the first database and the second database;

[0083] In an embodiment of the present invention, the first database may be a graph database (such as Nebula Graph, etc.), which may be used to process the storage, management, and query of graph data structures, etc.; the second database may be a distributed search and analysis engine, which may be used for full-text retrieval, structured retrieval, log analysis, and data indexing, etc. in a big data environment. During the data processing process, business data may be written to the second database in real time to support fast search, or business data may be written to the graph database to support complex relationship queries and analysis of relationships between different element objects and entities.

[0084] It should be noted that in order to improve the full-text search capability of the graph database under large amounts of data while ensuring data consistency, a multi-database multi-write mechanism is introduced. Based on Flink's real-time double-write method of the first database and the second database, the data consistency between the first database and the second database is guaranteed, ensuring the integrity and reliability of the business data, thereby improving the full-text search capability of the graph database under large amounts of data.

[0085] In a specific implementation, in a data processing flow based on a distributed stream processing framework, it is necessary to ensure connection management and data consistency in a concurrent environment. Therefore, it is necessary to first establish a connection between the distributed stream processing framework and the first database and the second database. Specifically, an initialization function for the first database and the second database may be determined first, so as to establish a connection between the distributed stream processing framework and the first database and the second database through the initialization function.

[0086] Among them, the initialization function is called during the initialization phase of the operator life cycle, allowing one-time initialization operations to be performed, such as connecting to external resources, opening database connections, or initializing some variables that need to be shared throughout the operator life cycle. The initialization function can ensure that the distributed stream processing framework can reliably interact with the first database and the second database during the processing of real-time data to ensure data consistency.

[0087] Step 102, initializing the first database and the second database according to the initialization function, and establishing connections between the distributed stream processing framework and the first database and the second database;

[0088] After the initialization function is determined, the distributed stream processing framework can initialize the first database and the second database respectively based on the initialization function, and establish a connection between the distributed stream processing framework and the first database and the second database, so as to ensure that the distributed stream processing framework can reliably interact with the first database and the second database during the process of processing real-time data to ensure data consistency.

[0089] In some feasible implementations, since the first database is a graph database, which may include a connection pool, the initialization function may include at least a connection pool initialization function corresponding to the connection pool. In the process of establishing a connection between the distributed stream processing framework and the first database and the second database, the connection pool can be initialized through the connection pool initialization function, and the connection pool can be set to an available state to establish a first connection between the first database and the distributed stream processing framework. At the same time, a database instance for the second database is obtained through the initialization function, and the second database is initialized according to the database instance, and a second connection between the second database and the distributed stream processing framework is established, thereby ensuring that the distributed stream processing framework can reliably interact with the first database and the second database in the process of processing real-time data to ensure data consistency.

[0090] In some feasible examples, for the first database and the second database, data can be processed through the corresponding clients, and the distributed stream processing framework can establish a connection corresponding to the client. Among them, the first database can be Nebula Graph, and the second database can be Elasticsearch. Then, the client can be initialized first to ensure that the Flink (i.e., distributed stream processing framework) job can reliably interact with Nebula Graph and Elasticsearch when processing real-time data, and ensure data consistency. Among them, Flink's open function (i.e., initialization function) is called in the initialization phase of the operator life cycle, allowing one-time initialization operations to be performed, such as connecting to external resources, opening database connections, or initializing some variables that need to be shared throughout the operator life cycle.

[0091] In the specific implementation, the client can be initialized in the open function of Flink, including the connection of Nebula client and ES client, ensuring that the two clients are initialized in the same open function to ensure the consistency of the connection. Among them, for the initialization of Nebula Graph, SessionPool can be used to manage the connection of NebulaGraph to ensure that the connection can be effectively managed in a high-concurrency environment. Then, the init() method is called in the open function to initialize the connection pool, thereby connecting Nebula Graph to Flink; for the initialization of Elasticsearch, RestHighLevelClient can be used to connect to the ES instance. Since the thread corresponding to the ES client is safe, it can be reused throughout the life cycle. Therefore, by initializing Nebula Graph and ES client in the same open function, it is ensured that the connection of both is successfully established at the beginning of the Flink task, thereby ensuring the stability of subsequent data processing. In addition, the release of resources can be ensured in the close function to avoid memory leaks or waste of connection pool resources.

[0092] Step 103, in response to receiving a data stream, the data stream includes at least one piece of service data;

[0093] After establishing a connection between the distributed stream processing framework and the first database and the second database, the data processing flow corresponding to the first database and the second database can be added to the stream processing flow corresponding to the distributed stream processing framework, so that after receiving the data stream, the distributed stream processing framework can process the received data stream, such as writing the data to the first database and the second database.

[0094] Among them, the data flow can be business-related data, such as user behavior data related to user behavior, etc. The data flow can include at least one business data. In the process of processing the data, it can be processed with a single business data as the processing dimension to ensure the consistency and integrity of data processing, while reducing the amount of data processed, which can effectively improve data processing efficiency.

[0095] Step 104, extracting at least one key field for identifying the business data from the business data, and determining a data identifier corresponding to the business data according to the at least one key field;

[0096] During the data processing process, the distributed stream processing framework can extract at least one key field for identifying the business data from the business data, and determine the data identifier corresponding to the business data based on at least one key field, so as to ensure the uniqueness of the data by generating a unique data identifier corresponding to the business data, reduce duplicate writing, and ensure the consistency and integrity of the data between the first database and the second database.

[0097] In some feasible implementations, at least one key field may be concatenated into a string corresponding to the business data, and then the string may be converted into a data identifier that matches the business data. Wherein, the key field includes at least one of a user identifier, a timestamp, and an event type, and then the user identifier, the timestamp, and the event type may be concatenated into a string corresponding to the business data. Optionally, the string may be used as an input of a hash function, and the string may be operated to obtain a hash value that matches the business data, and the hash value may be used as the data identifier of the business data.

[0098] In one example, for the data processing process, the consistent writing of data can be achieved through Flink's processFunction function. Before the data is officially written, the data identifier corresponding to the business data can be generated first. For each piece of business data received, the MD5 (Message-Digest Algorithm) unique ID corresponding to each piece of business data can be generated first to ensure the idempotence of the data, that is, when the same data is written multiple times, the results are the same, thereby avoiding data duplication or loss.

[0099] In the specific implementation, for each piece of business data received, Flink's processFunction can process the input data stream, processing one piece of data at a time. Among them, each piece of data may be an object containing multiple fields, such as a record or a message, and the key field used to generate a unique ID can be extracted from the received data. The selected key field is the field that uniquely identifies the business data, such as user ID, timestamp, event type, etc. Then, these fields can be concatenated into a string as the input for generating MD5. In this process, the order and format of the fields can be fixed to prevent the generated ID from being inconsistent. Then, the MessageDigest class in Java is used to calculate the MD5 hash value, and the hash value is used as the unique ID of this business data, thereby ensuring the uniqueness of the data by generating a unique data identifier corresponding to the business data, reducing duplicate writing, and ensuring the consistency and integrity of the data between the first database and the second database.

[0100] Step 105: write the business data into the first database according to the data identifier, and write the business data into the second database if the business data is successfully written into the first database.

[0101] In the process of writing data into the database, on the one hand, the data can be written to the first database and the second database according to the data identifier to achieve consistent data writing. On the other hand, the consistency of data writing can be further ensured by first writing to the first database and then writing to the second database.

[0102] In some feasible implementations, the distributed stream processing framework can first write the business data into the first database according to the data identifier, and detect whether the business data is successfully written into the first database. If it is detected that the business data fails to be written into the first database, the distributed stream processing framework is triggered to rewrite the business data into the first database according to the data identifier until the business data is successfully written into the first database, and then write the business data into the second database. If it is detected that the business data is successfully written into the first database, the business data is written into the second database according to the data identifier, thereby ensuring the uniqueness of the data by generating a unique data identifier corresponding to the business data, reducing duplicate writing, and ensuring the consistency and integrity of the data between the first database and the second database.

[0103] Correspondingly, in the process of writing to the second database, it is also possible to detect whether the business data is successfully written to the second database. If it is detected that the business data fails to be written to the second database, the distributed stream processing framework is triggered to rewrite the business data to the second database according to the data identifier. Therefore, in the process of data writing, when data writing fails, Flink can automatically trigger a retry to ensure that the data written to each database is consistent.

[0104] In one example, as described in the previous embodiment, data can be written to Nebula Graph first, and exceptions can be caught. If the write fails, an exception is thrown to trigger Flink's retry mechanism. If the data is successfully written to Nebula Graph, the same data is written to Elasticsearch again, and the exception is caught. Similarly, if writing to Elasticsearch fails, Flink will automatically retry to ensure data consistency. In this process, since a unique data identifier is generated through MD5, it can be effectively ensured that each write operation is idempotent. In the event of a write failure, the MD5 ID can be used to rewrite the data, overwriting the previous erroneous data, thereby ensuring the consistency of data writing.

[0105] In some feasible implementations, for business data written into the first database and the second database, when performing data query, the distributed stream processing framework can respond to the data query instruction for the second database, query the target business data corresponding to the data query instruction in the second database, and if the target business data is queried in the second database, the target business data returned by the second database is received; if the target business data is not queried in the second database, the target business data corresponding to the data query instruction can be queried in the first database; if the target business data is queried in the first database, the target business data returned by the first database is received; if the target business data is not queried in the first database, a failure prompt message for the data query instruction is output, and the failure prompt message is used to prompt that the relevant data is not queried, thereby effectively improving the full-text retrieval capability of the graph database under large data volumes by writing the same business data in multiple databases.

[0106] For example, assuming that the first database is Nebula Graph and the second database is Elasticsearch, when performing data query, Flink can first query whether the corresponding business data exists in Elasticsearch. If so, it can directly return the corresponding data; if not, Flink can query whether the corresponding business data exists in Nebula Graph. If so, it returns the corresponding data; if not, it prompts that the corresponding data does not exist. In this way, by writing the same business data in multiple databases, the full-text retrieval capability of the graph database under large data volumes is effectively improved.

[0107] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0108] In an embodiment of the present invention, it is applied to a distributed stream processing framework. In the process of processing business data, an initialization function for a first database and a second database may be determined first, and then the first database and the second database may be initialized according to the initialization function to establish a connection between the distributed stream processing framework and the first database and the second database. By initializing the first database and the second database in an initialization function, a corresponding connection is established to ensure the consistency of the connection. In the process of writing data, in response to receiving a data stream, the data stream includes at least one business data, and then at least one key field for identifying the business data is extracted from the business data, and a data identifier corresponding to the business data is determined according to at least one key field. Then, the business data is written to the first database according to the data identifier, and when the first database is successfully written, the business data is written to the second database. By generating a unique data identifier corresponding to the business data, the uniqueness of the data is ensured, and while reducing duplicate writing, the consistency and integrity of the data between the first database and the second database are ensured.

[0109] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0110] In Flink jobs, by initializing Nebula Graph and Elasticsearch clients simultaneously in the open function, and using MD5 hashing to generate a unique ID for each piece of data, the data processing flow of first writing to NebulaGraph and then writing to Elasticsearch is implemented in processFunction. This process ensures connection management and data consistency in a concurrent environment. At the same time, by using the idempotence of MD5 ID, it can safely retry when writing fails and overwrite erroneous data, thereby maintaining data integrity and accuracy. Figure 2 , shows a schematic diagram of the initialization process provided in an embodiment of the present invention, and the specific implementation steps are as follows:

[0111] 1. Initialize the client

[0112] First, initialize the client to ensure that the Flink job can reliably interact with NebulaGraph and Elasticsearch and ensure data consistency when processing real-time data.

[0113] Flink's open function is called during the initialization phase of the operator life cycle, allowing one-time initialization operations to be performed, such as connecting to external resources, opening a database connection, or initializing some variables that need to be shared throughout the operator life cycle.

[0114] Initialization method: In Flink's open function, you can initialize the client. The initialization content includes the connection between the Nebula client and the ES client. Ensure that the two clients are initialized in the same open function to ensure connection consistency.

[0115] (I) Nebula Graph initialization:

[0116] Use SessionPool to manage Nebula Graph connections to ensure effective connection management in a high-concurrency environment. Then call the init() method in open to initialize the connection pool.

[0117] (II) Elasticsearch initialization:

[0118] Use RestHighLevelClient to connect to the ES instance. This client is thread-safe and can be reused throughout the life cycle.

[0119] (III) Connection consistency:

[0120] By initializing Nebula Graph and ES client in the same open function, we ensure that the connection between them is successfully established when the Flink task starts, thus ensuring the stability of subsequent data processing.

[0121] Ensure the release of resources in the close function to avoid memory leaks or waste of connection pool resources.

[0122] 2. Data processing

[0123] For data processing, refer to Figure 3 , which shows a schematic diagram of the data writing process provided in an embodiment of the present invention. In the processFunction function of Flink, data consistency writing can be achieved through the following steps:

[0124] (I) Receive a piece of data and generate an MD5 unique ID

[0125] The step of generating an MD5 unique ID is crucial because it is used to ensure the idempotence of the data, that is, when the same data is written multiple times, the results are the same, thus avoiding data duplication or loss.

[0126] 1) Receive a piece of data:

[0127] Flink's processFunction processes the input data stream, one piece of data at a time. Usually, this piece of data may be an object with multiple fields, such as a record or a message.

[0128] 2) Extract key fields:

[0129] From the received data, extract the key fields used to generate a unique ID. The selected fields should be able to uniquely identify the data, such as user ID, timestamp, event type, etc.

[0130] These fields can be concatenated into a string as input to generate the MD5. Make sure the order and format of the fields are fixed to prevent inconsistent generated IDs.

[0131] 3) Generate MD5 hash value:

[0132] Use the MessageDigest class in Java to calculate the MD5 hash value. This hash value will serve as the unique ID of this data. The main feature of MD5 is that it generates the same 128-bit hash value (usually expressed as a 32-bit hexadecimal string) for the same input.

[0133] (2) Write to Nebula Graph first, then write to ES.

[0134] First, write the data to Nebula Graph and catch the exception. If the write fails, throw an exception to trigger Flink's retry mechanism.

[0135] If the data is successfully written to Nebula Graph, the same data is written to Elasticsearch and the exception is caught.

[0136] (3) If writing fails, Flink will automatically retry to ensure data consistency.

[0137] 3. Ensure idempotent writes

[0138] The unique ID generated by MD5 ensures that each write operation is idempotent. If the write fails, you can rewrite it using the MD5 ID to overwrite the previous erroneous data.

[0139] The above steps ensure the consistency of data when it is written to Nebula Graph and Elasticsearch, and Flink's retry mechanism ensures the consistency of the final data when a failure occurs.

[0140] In addition, refer to Figure 4, showing a schematic diagram of data query provided in an embodiment of the present invention. When performing data query, Flink can first query whether the corresponding business data exists in Elasticsearch (ie ES). If so, the corresponding data can be directly returned; if not, Flink can query whether the corresponding business data exists in Nebula Graph. If so, the corresponding data is returned; if not, a prompt is given that the corresponding data does not exist. Thus, by writing the same business data in multiple databases, the full-text retrieval capability of the graph database under large data volumes is effectively improved.

[0141] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0142] Reference Figure 5 , shows a structural block diagram of a data processing device provided in an embodiment of the present invention, involving a first database and a second database, wherein the first database is at least used for processing the storage, management and query of graph data structures, and the second database is at least used for full-text retrieval, structured retrieval, log analysis and data indexing in a big data environment; wherein the processing device is applied to a distributed stream processing framework, and may specifically include the following modules:

[0143] A function determination module 501, configured to determine initialization functions for the first database and the second database;

[0144] An initialization module 502, configured to initialize the first database and the second database according to the initialization function, and establish a connection between the distributed stream processing framework and the first database and the second database;

[0145] A data receiving module 503, configured to respond to receiving a data stream, wherein the data stream includes at least one piece of service data;

[0146] An identification generating module 504, configured to extract at least one key field for identifying the business data from the business data, and determine a data identification corresponding to the business data according to the at least one key field;

[0147] The data writing module 505 is used to write the business data into the first database according to the data identifier, and write the business data into the second database if the business data is successfully written into the first database.

[0148] In some feasible implementations, the first database includes a connection pool, the initialization function includes at least a connection pool initialization function corresponding to the connection pool, and the initialization module 502 is specifically used to:

[0149] Initialize the connection pool through the connection pool initialization function, set the connection pool to an available state, and establish a first connection between the first database and the distributed stream processing framework;

[0150] A database instance for the second database is obtained through the initialization function, and the second database is initialized according to the database instance, so as to establish a second connection between the second database and the distributed stream processing framework.

[0151] In some feasible implementations, the identification generation module 504 is specifically used to:

[0152] Concatenate at least one of the key fields into a character string corresponding to the business data;

[0153] The character string is converted into a data identifier that matches the business data.

[0154] In some feasible implementations, the key field includes at least one of a user identifier, a timestamp, and an event type, and the identifier generation module 504 is specifically used to:

[0155] The user identifier, the timestamp, and one of the event types are concatenated into a character string corresponding to the business data.

[0156] In some feasible implementations, the identification generation module 504 is specifically used to:

[0157] The character string is used as an input of a hash function, an operation is performed on the character string to obtain a hash value matching the business data, and the hash value is used as a data identifier of the business data.

[0158] In some feasible implementations, the data writing module 505 is specifically used for:

[0159] Writing the business data into the first database according to the data identifier;

[0160] If it is detected that the business data fails to be written into the first database, the distributed stream processing framework is triggered to re-write the business data into the first database according to the data identifier;

[0161] If it is detected that the business data is successfully written into the first database, the business data is written into the second database according to the data identifier.

[0162] In some feasible implementations, the data writing module 505 is specifically used for:

[0163] If it is detected that the business data fails to be written into the second database, the distributed stream processing framework is triggered to re-write the business data into the second database according to the data identifier.

[0164] Some possible implementations also include:

[0165] A first query module, configured to, in response to a data query instruction for the second database, query the second database for target business data corresponding to the data query instruction;

[0166] a first data returning module, configured to receive the target business data returned by the second database if the target business data is found in the second database;

[0167] A second query module, configured to query the first database for the target business data corresponding to the data query instruction if the target business data is not found in the second database;

[0168] A second data returning module, configured to receive the target business data returned by the first database if the target business data is found in the first database;

[0169] A prompt module is used to output failure prompt information for the data query instruction if the target business data is not found in the first database, and the failure prompt information is used to prompt that the relevant data is not found.

[0170] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0172] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0173] Figure 6 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0174] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and a power supply 611. Those skilled in the art will appreciate that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0175] It should be understood that in the embodiment of the present invention, the radio frequency unit 601 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 610 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 601 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 601 can also communicate with the network and other devices through a wireless communication system.

[0176] The electronic device provides users with wireless broadband Internet access through the network module 602, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0177] The audio output unit 603 can convert the audio data received by the RF unit 601 or the network module 602 or stored in the memory 609 into an audio signal and output it as sound. Moreover, the audio output unit 603 can also provide audio output related to a specific function performed by the electronic device 600 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 603 includes a speaker, a buzzer, a receiver, etc.

[0178] The input unit 604 is used to receive audio or video signals. The input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 606. The image frame processed by the graphics processor 6041 can be stored in the memory 609 (or other storage medium) or sent via the radio frequency unit 601 or the network module 602. The microphone 6042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 601 in the case of a telephone call mode.

[0179] The electronic device 600 also includes at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 6061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 6061 and / or the backlight when the electronic device 600 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 605 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0180] The display unit 606 is used to display information input by the user or information provided to the user. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0181] The user input unit 607 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 607 includes a touch panel 6071 and other input devices 6072. The touch panel 6071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 6071 or near the touch panel 6071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 6071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 610, receives the command sent by the processor 610 and executes it. In addition, the touch panel 6071 can be implemented in various types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 6071, the user input unit 607 may also include other input devices 6072. Specifically, other input devices 6072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0182] Further, the touch panel 6071 may be covered on the display panel 6061. When the touch panel 6071 detects a touch operation on or near it, it is transmitted to the processor 610 to determine the type of the touch event, and then the processor 610 provides a corresponding visual output on the display panel 6061 according to the type of the touch event. It can be understood that in one embodiment, the touch panel 6071 and the display panel 6061 are used as two independent components to implement the input and output functions of the electronic device, but in some embodiments, the touch panel 6071 and the display panel 6061 can be integrated to implement the input and output functions of the electronic device, which is not limited here.

[0183] The interface unit 608 is an interface for connecting an external device to the electronic device 600. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 608 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 600 or may be used to transmit data between the electronic device 600 and an external device.

[0184] The memory 609 can be used to store software programs and various data. The memory 609 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 609 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0185] The processor 610 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 609 and calling data stored in the memory 609, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 610 may include one or more processing units; preferably, the processor 610 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 610.

[0186] The electronic device 600 may also include a power supply 611 (such as a battery) for supplying power to each component. Preferably, the power supply 611 may be logically connected to the processor 610 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0187] In addition, the electronic device 600 includes some functional modules not shown, which will not be described in detail here.

[0188] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0190] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0191] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0192] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0193] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0196] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0197] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: Involving a first database and a second database, the first database is at least used for processing the storage, management and query of graph data structures, and the second database is at least used for full-text retrieval, structured retrieval, log analysis and data indexing in a big data environment; wherein the processing method is applied to a distributed stream processing framework, including: Determine initialization functions for the first database and the second database; Initialize the first database and the second database according to the initialization function, and establish connections between the distributed stream processing framework and the first database and the second database; In response to receiving a data stream, the data stream includes at least one piece of service data; Extracting at least one key field for identifying the business data from the business data, and determining a data identifier corresponding to the business data according to the at least one key field; The business data is written into the first database according to the data identifier, and if the business data is successfully written into the first database, the business data is written into the second database.

2. The method according to claim 1, characterized in that The first database includes a connection pool, the initialization function includes at least a connection pool initialization function corresponding to the connection pool, and the first database and the second database are initialized according to the initialization function to establish a connection between the distributed stream processing framework and the first database and the second database, including: Initialize the connection pool through the connection pool initialization function, set the connection pool to an available state, and establish a first connection between the first database and the distributed stream processing framework; A database instance for the second database is obtained through the initialization function, and the second database is initialized according to the database instance, so as to establish a second connection between the second database and the distributed stream processing framework.

3. The method according to claim 1, characterized in that The determining, according to at least one of the key fields, a data identifier corresponding to the business data includes: Concatenate at least one of the key fields into a character string corresponding to the business data; The character string is converted into a data identifier that matches the business data.

4. The method according to claim 3, characterized in that The key field includes at least one of a user identifier, a timestamp, and an event type, and the step of concatenating at least one of the key fields into a character string corresponding to the business data includes: The user identifier, the timestamp, and one of the event types are concatenated into a character string corresponding to the business data.

5. The method according to claim 3 or 4, characterized in that: The converting the character string into a data identifier matching the business data includes: The character string is used as an input of a hash function, an operation is performed on the character string to obtain a hash value matching the business data, and the hash value is used as a data identifier of the business data.

6. The method according to claim 1, characterized in that The step of writing the business data into the first database according to the data identifier, and writing the business data into the second database if the business data is successfully written into the first database, comprises: Writing the business data into the first database according to the data identifier; If it is detected that the business data fails to be written into the first database, the distributed stream processing framework is triggered to re-write the business data into the first database according to the data identifier; If it is detected that the business data is successfully written into the first database, the business data is written into the second database according to the data identifier.

7. The method according to claim 6, characterized in that Writing the business data into the second database according to the data identifier includes: If it is detected that the business data fails to be written into the second database, the distributed stream processing framework is triggered to re-write the business data into the second database according to the data identifier.

8. The method according to any one of claims 1 to 7, characterized in that: Also includes: In response to a data query instruction for the second database, querying the second database for target business data corresponding to the data query instruction; If the target business data is found in the second database, receiving the target business data returned by the second database; If the target business data is not found in the second database, searching the first database for the target business data corresponding to the data query instruction; If the target service data is found in the first database, receiving the target service data returned by the first database; If the target business data is not found in the first database, a failure prompt message for the data query instruction is output, where the failure prompt message is used to prompt that the relevant data is not found.

9. A data processing device, characterized in that: Involving a first database and a second database, the first database is at least used for processing the storage, management and query of graph data structures, and the second database is at least used for full-text retrieval, structured retrieval, log analysis and data indexing in a big data environment; wherein the processing device is applied to a distributed stream processing framework, including: A function determination module, used to determine initialization functions for the first database and the second database; An initialization module, configured to initialize the first database and the second database according to the initialization function, and establish a connection between the distributed stream processing framework and the first database and the second database; A data receiving module, configured to respond to receiving a data stream, wherein the data stream includes at least one piece of service data; an identification generating module, used to extract at least one key field for identifying the business data from the business data, and determine a data identification corresponding to the business data according to at least one key field; A data writing module is used to write the business data into the first database according to the data identifier, and write the business data into the second database if the business data is successfully written into the first database.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 8 when executing the program stored in the memory.

11. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 8.

Images