A data operation method and system based on a data processing engine

Abstract

The application provides a data operation method and system based on a data processing engine, and the method comprises the following steps: in the data processing engine, configuration start configuration information is configured; the configuration start configuration information at least comprises a schema file of a target topic data group in an open source stream processing platform; the schema file is subjected to deserialization processing to obtain the target topic data group; a listening thread is created according to the target topic data group; when the listening thread listens to to-be-written data of the open source stream processing platform, a data writing statement of HBase is generated according to a pre-created HBase data table template; and finally, the data writing statement of HBase is written into an HBase database. The data processing engine can be used to avoid the direct operation of a developer on multi-source data, thereby reducing the development complexity of multi-source data storage and the cost of multi-source data maintenance, and being beneficial to improving the research and development efficiency and research and development quality of the developer.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a data manipulation method and system based on a data processing engine. Background Technology

[0002] Currently, security big data analytics systems collect all audit logs and traffic logs during daily enterprise security operations. However, in real-world environments, the total amount of audit logs and traffic logs collected daily by these systems can reach hundreds of gigabytes or even several terabytes. The industry typically employs multi-source storage to reduce data footprint while meeting analytical needs.

[0003] Both HBase and Elasticsearch are distributed and scalable storage media, effectively supporting multi-source data storage. However, each has its own strengths: HBase excels at data storage and compression, while Elasticsearch excels at flexible data retrieval. To leverage their respective advantages, the industry often combines them, storing search terms used for data analysis in Elasticsearch and raw data in HBase, thus achieving low-cost storage with flexible retrieval.

[0004] However, adopting this approach would require developers to operate on data using multiple different storage media (ElasticSearch and HBase) during daily development, which would greatly complicate data operations and lead to a certain degree of loss in development efficiency and functional quality. Summary of the Invention

[0005] The purpose of this application is to provide a data operation method and system based on a data processing engine, which can use the data processing engine to avoid researchers directly manipulating multi-source data, thereby reducing the development complexity and maintenance cost of multi-source data storage, and improving the research and development efficiency and quality of researchers.

[0006] The first aspect of this application provides a data manipulation method based on a data processing engine, including:

[0007] In the data processing engine, startup configuration information is configured; the startup configuration information includes at least an outline file of the target topic data group in the open-source stream processing platform;

[0008] The outline document is deserialized to obtain the target topic data set;

[0009] Create a listening thread based on the target topic data group;

[0010] When the listening thread detects the data to be written from the open-source stream processing platform, it generates an HBase data write statement based on the pre-created HBase data table template.

[0011] The data write statement in HBase is written to the HBase database.

[0012] In the above implementation, the data processing engine is configured with startup configuration information. This startup configuration information includes at least an outline file of the target topic data group in the open-source stream processing platform. The outline file is deserialized to obtain the target topic data group. A listening thread is then created based on the target topic data group. When the listening thread detects data to be written from the open-source stream processing platform, an HBase data write statement is generated based on a pre-created HBase data table template. Finally, the HBase data write statement is written to the HBase database. This allows the data processing engine to avoid direct manipulation of multi-source data by developers, thereby reducing the development complexity and maintenance costs of multi-source data storage, and improving the development efficiency and quality of developers.

[0013] Furthermore, the step of generating an HBase data write statement based on a pre-created HBase data table template when the listening thread detects data to be written from the open-source stream processing platform includes:

[0014] When the listening thread detects the data to be written from the open-source stream processing platform, it generates a unique data identifier that matches the data to be written.

[0015] Based on the unique data identifier and the pre-created HBase data table template, an HBase data write statement is generated; the HBase data write statement includes the unique data identifier.

[0016] Furthermore, after the step of writing the HBase data write statement to the HBase database, the method further includes:

[0017] The HBase scanner is started via a MapReduce task;

[0018] According to a preset interval, the scanner cyclically scans the latest unique identifier of the data in the HBase;

[0019] Determine whether the latest data unique identifier is greater than the last data unique identifier; the last data unique identifier is the latest data unique identifier detected by the scanner in the last scan.

[0020] When the latest data unique identifier is greater than the last data unique identifier, obtain the latest HBase data that matches the latest data unique identifier;

[0021] Convert the latest HBase data into structured code data;

[0022] The code data is written to Elasticsearch as Elasticsearch data, and an Elasticsearch index is generated based on the latest data's unique identifier and a pre-created Elasticsearch index template.

[0023] Furthermore, the method also includes:

[0024] The data lifecycle management thread retrieves Elasticsearch storage data in Elasticsearch in a loop according to a preset interval.

[0025] Based on a preset time management strategy or a preset space management strategy, determine whether the Elasticsearch stored data has expired or whether the storage space occupied by the Elasticsearch stored data is too large.

[0026] When the Elasticsearch stored data expires or the storage space occupied by the Elasticsearch stored data becomes too large, delete the Elasticsearch index of the Elasticsearch data with the smallest unique identifier matched by the daily prefix.

[0027] Furthermore, the method also includes:

[0028] Receive SPL search statements input by the user;

[0029] The SPL retrieval statement is parsed to obtain a statement queue containing multiple retrieval sub-statements;

[0030] Each retrieval sub-statement in the statement queue is converted into an Elasticsearch query statement;

[0031] Based on the query statement, call the Elasticsearch data manipulation interface to perform the query and obtain the Elasticsearch query results;

[0032] Based on the Elasticsearch query results, obtain the HBase query results;

[0033] The Elasticsearch query results and the HBase query results are merged to obtain the SPL retrieval results.

[0034] A second aspect of this application provides a data operating system based on a data processing engine, the data operating system based on a data processing engine comprising:

[0035] A configuration unit is used to configure startup configuration information in the data processing engine; the startup configuration information includes at least an outline file of the target topic data group in the open-source stream processing platform.

[0036] A processing unit is used to deserialize the outline document to obtain the target topic data group;

[0037] A creation unit is used to create a listening thread based on the target topic data group;

[0038] The generation unit is used to generate HBase data writing statements based on a pre-created HBase data table template when the listening thread detects the data to be written from the open-source stream processing platform.

[0039] The write unit is used to write the data write statement of the HBase database.

[0040] In the above implementation, the configuration unit configures startup configuration information in the data processing engine; the startup configuration information includes at least an outline file of the target topic data group in the open-source stream processing platform; the processing unit deserializes the outline file to obtain the target topic data group; the creation unit then creates a listening thread based on the target topic data group; when the listening thread detects the data to be written from the open-source stream processing platform, the generation unit generates an HBase data write statement based on a pre-created HBase data table template; finally, the writing unit writes the HBase data write statement to the HBase database. This allows the data processing engine to avoid direct manipulation of multi-source data by developers, thereby reducing the development complexity and maintenance cost of multi-source data storage, and improving the development efficiency and quality of developers.

[0041] Furthermore, the generation unit is specifically used to generate a unique data identifier that matches the data to be written when the listening thread detects the data to be written from the open-source stream processing platform; and to generate an HBase data write statement based on the unique data identifier and a pre-created HBase data table template; the HBase data write statement includes the unique data identifier.

[0042] Furthermore, the data operating system also includes:

[0043] The startup unit is used to start the HBase scanner via a MapReduce task after the step of writing the data write statement of the HBase to the HBase database;

[0044] The monitoring unit is used to cyclically scan the latest data unique identifier in the HBase according to a preset interval.

[0045] The judgment unit is used to determine whether the latest data unique identifier is greater than the last data unique identifier; the last data unique identifier is the latest data unique identifier scanned by the scanner in the last scan.

[0046] The acquisition unit is used to acquire the latest HBase data that matches the latest data unique identifier when the latest data unique identifier is greater than the last data unique identifier;

[0047] The generation unit is further configured to convert the latest HBase data into structured code data; write the code data as Elasticsearch data into Elasticsearch; and generate an Elasticsearch index based on the unique identifier of the latest data and a pre-created Elasticsearch index template.

[0048] A third aspect of this application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform the data manipulation method based on a data processing engine as described in any one of the first aspects of this application.

[0049] The fourth aspect of this application provides a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the data manipulation method based on a data processing engine as described in any one of the first aspects of this application. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating a data manipulation method based on a data processing engine, provided for an embodiment of this application;

[0052] Figure 2A schematic diagram of the structure of a data operating system based on a data processing engine is provided in an embodiment of this application;

[0053] Figure 3 This is a flowchart illustrating a data operation method based on a data processing engine for secure data storage across storage media, as provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0055] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0056] Example 1

[0057] Please refer to Figure 1 , Figure 1 This embodiment provides a flowchart illustrating a data manipulation method based on a data processing engine. The data manipulation method based on the data processing engine includes:

[0058] S101. In the data processing engine, configure the startup configuration information; the startup configuration information shall include at least the outline file of the target topic data group in the open source stream processing platform.

[0059] In this embodiment, the open-source stream processing platform can specifically be Kafka, an open-source stream processing platform developed by the Apache Software Foundation. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data from consumers on a website.

[0060] In this embodiment of the application, the outline file is specifically a schema file, so the startup configuration information includes at least the schema file of the topic in Kafka. The schema file, namely the Avro schema file, defines the structure of the schema during data serialization and deserialization.

[0061] Please refer to the following: Figure 3 , Figure 3 This is a flowchart illustrating a data operation method based on a data processing engine for secure data storage across storage media, as provided in an embodiment of this application.

[0062] like Figure 3As shown, this method is applied to a data processing engine, which is a pluggable plugin that works in conjunction with a secure big data analytics system. It requires configuration of startup information, including a schema file (AVRO serialization), connection information for HBase and Elasticsearch, the lifecycle of stored data, and the time-series field name for data deletion. The schema file stores the Kafka connection information and topic of the source data. The lifecycle includes a time strategy (hours) or a space strategy (MB). The time-series field name for data deletion defaults to deleting the earliest data.

[0063] AVRO refers to a data serialization system with rich data structure types and fast, compressible binary data formats.

[0064] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.

[0065] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.

[0066] In this embodiment of the application, the method can be applied to a storage architecture that combines ElasticSearch and HBase to store massive amounts of secure data in a secure big data system while saving storage costs.

[0067] For example, the startup configuration information in the data processing engine includes configuring HBase-site.xml and Elasticsearch connection information. An example of configuring HBase-site.xml is as follows:

[0068]

[0069]

[0070] Here is an example of configuring Elasticsearch connection information:

[0071] SPL.Elasticsearch.address=127.0.0.1:23200

[0072] SPL.Kafka.address=127.0.0.1:19092

[0073] SPL.Kafka.schema=alarm_event.avsc

[0074] SPL.data.lifeCycle.hours=168

[0075] SPL.data.lifeCycle.Mbytes=8096

[0076] SPL.delete field = time

[0077] S102. Deserialize the outline document to obtain the target topic data set.

[0078] In this embodiment of the application, the XSL stylesheet consists of one or more sets of rules called templates.

[0079] In this embodiment of the application, the method is to deserialize the topic name and metadata according to the configured schema file, and then create the HBase data table and the template for the index in Elasticsearch (the template is not written to the source and the template needs to add a rowkey field).

[0080] In this embodiment of the application, the rowkey field corresponds to the unique identifier of the subsequent data.

[0081] S103. Create a listening thread based on the target topic data group.

[0082] S104. When the listening thread detects data to be written from the open-source stream processing platform, it generates a unique data identifier that matches the data to be written.

[0083] In this embodiment of the application, the unique identifier of the data is a combination of the timestamp and the MD5 hash of the data.

[0084] S105. Generate HBase data write statements based on the data unique identifier and the pre-created HBase data table template; the HBase data write statements include the data unique identifier.

[0085] In this embodiment of the application, after the system starts successfully, the data processing engine will create (the same number of topics but no more than the number of CPU cores / 2) threads to connect to the Kafka data source, listen to the data of the topics, and when new data is generated, multiple threads will persist the new data to HBase.

[0086] In this embodiment, the specific approach is as follows: after a thread consumes data, it generates a unique value (timestamp_(src.MD5)) for each data record, where timestamp is the timestamp of the current machine (accurate to microseconds) and src is the original log; then the unique value is used as the rowkey for each data record to be written to HBase; and then the data is written to HBase in batches.

[0087] In this embodiment, HBase is a distributed, column-oriented open-source database and a distributed storage system for structured data.

[0088] In this embodiment of the application, the schema file is exemplified as follows:

[0089]

[0090]

[0091] Here, `name` is the name of the topic in Kafka, and also the name of the data table in HBase and Elasticsearch. `fields` contains the field information in the data table, `type` is the field type, and `default` is the default value of the field.

[0092] In this embodiment of the application, the example of the table creation statement for creating HBase is as follows:

[0093] create'alarm_event,'DVC_T','DVC_ID'

[0094] The corresponding template statement in creating an Elasticsearch instance is as follows:

[0095]

[0096]

[0097] S106. Write the HBase data to the HBase database using the write statement.

[0098] In this embodiment of the application, implementing the above steps S101 to S106 can solve the problem that when massive amounts of secure data are stored on multiple storage media, it is necessary to operate multiple different storage media (including the index library ElasticSearch and the database HBase) simultaneously to complete the data writing, while also dealing with the data consistency problem, thereby reducing the complexity of data writing and improving data writing efficiency.

[0099] As an optional implementation, writing data to the HBase database using HBase write statements includes the following steps:

[0100] Create a thread to connect to the Kafka data source;

[0101] Listen to data in a topic from a Kafka data source;

[0102] When a new HBase data write statement is detected, multiple threads are started to persist the HBase data write statement to the HBase database.

[0103] In the above implementation, the number of threads created by the data processing engine is the same as the number of topics, but does not exceed the number of CPU cores / 2.

[0104] In the above implementation, after the thread listens for data, it generates a unique value for each piece of data (timestamp_(src.MD5)), where timestamp is the timestamp of the current machine (accurate to microseconds) and src is the original log; then the unique value is used as the rowkey for each piece of data to be written to HBase; and then the data is written to HBase in batches.

[0105] As an optional implementation, after writing the HBase data write statement to the HBase database, the method further includes:

[0106] Start the HBase scanner using a MapReduce task;

[0107] Based on a preset interval, the scanner continuously scans for the latest unique identifier of data in HBase.

[0108] Determine if the unique identifier of the latest data is greater than the unique identifier of the last data; the unique identifier of the last data is the unique identifier of the latest data last scanned by the scanner;

[0109] When the latest data unique identifier is greater than the last data unique identifier, retrieve the latest HBase data that matches the latest data unique identifier.

[0110] Convert the latest HBase data into structured code data;

[0111] Write code data as Elasticsearch data to Elasticsearch and generate an Elasticsearch index based on the latest data unique identifier and a pre-created Elasticsearch index template.

[0112] In the above implementation, the latest HBase data is converted into structured code data, that is, the latest HBase data is converted into Elasticsearch write statements, and the structured code data is the Elasticsearch write statement.

[0113] In the above implementation, the latest HBase data can be obtained based on the latest data unique identifier.

[0114] In the above implementation, the HBase scanner is specifically the HBase Scanner.

[0115] In the above implementation, MapReduce is a programming model for parallel computation of large-scale datasets (greater than 1TB). It greatly facilitates programmers in running their programs on distributed systems even without prior knowledge of distributed parallel programming.

[0116] In the above implementation, the code data can specifically be JSON data. JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is based on a subset of ECMAScript (the JavaScript specification developed by the European Computer Association) and uses a text format that is completely independent of programming languages ​​to store and represent data.

[0117] In the above implementation, after the data processing engine starts, it will start a MapReduce task for data synchronization. This task will start an HBase Scanner to periodically scan the changed data in HBase and write the changed data to Elasticsearch in batches.

[0118] In the above implementation, after the HBase Scanner starts, it scans data with an lexicographical order greater than the rowkey of the last scan. Then, it converts the scanned data into a structured JSON format and records the rowkey of each data item (adding it to the structured JSON data). Then, it writes the data in batches to Elasticsearch. The indexes in Elasticsearch are created in chronological order, with the index created on the current day, and the rowkey of the last data item is recorded.

[0119] In the above implementation, the above steps can be repeated periodically to write the changed data to Elasticsearch in batches.

[0120] In this embodiment of the application, the process of writing data to HBase is exemplified as follows:

[0121] put'alarm_event','1648372839000000absczdrfs','alarm_event:DVC_T','x123'

[0122] put'alarm_event','1648372839000000absczdrfd','alarm_event:DVC_ID','123'

[0123] In this embodiment of the application, the data written to Elasticsearch is as follows:

[0124]

[0125] As an optional implementation, the method may further include:

[0126] The data lifecycle management thread retrieves Elasticsearch storage data in Elasticsearch in a loop according to a preset interval.

[0127] Based on a preset time management strategy or a preset space management strategy, determine whether the Elasticsearch stored data has expired or whether the storage space occupied by the Elasticsearch stored data is too large.

[0128] When Elasticsearch stored data expires or the storage space occupied by the Elasticsearch stored data becomes too large, delete the Elasticsearch index of the Elasticsearch data with the smallest unique identifier matched by the daily prefix.

[0129] In the above implementation, the unique data identifier is a timestamp format, which is a prefix generated by the current day's timestamp. First, the unique data identifier matching the daily prefix is ​​determined, and then the data that matches the prefix is ​​deleted.

[0130] In this embodiment, Elasticsearch is a search server based on Lucene. It provides a distributed, multi-user capable full-text search engine based on a RESTful web interface. Elasticsearch is developed in Java and released as open source under the Apache License; it is a popular enterprise-grade search engine.

[0131] In this embodiment, it is determined whether the data stored in Elasticsearch has expired or whether the storage space occupied by the data stored in Elasticsearch is too large. That is, it is determined whether the data stored in the current medium has reached the data deletion threshold. If the threshold is reached, the data deletion action is triggered. When deleting data, the program will first delete the earliest index in Elasticsearch. Since the rowkey in HBase is created in time order, it can continue to scan the data in HBase that is younger than the current smallest time in Elasticsearch.

[0132] In this embodiment of the application, implementing the above steps can solve the problem of data inconsistency caused by researchers needing to operate two different storage media simultaneously when managing the lifecycle of massive amounts of data.

[0133] In this embodiment of the application, implementing steps S101 to S106 above enables data writing to the data processing engine. For example... Figure 3 As shown, first, create the HBase data table template and the Elasticsearch index template. When the secure big data analytics system starts, it deserializes the topic name and metadata based on the configured schema file, and then creates the HBase data table and the Elasticsearch index template. The template does not write to the source, and a rowkey field is added to the template. The rowkey is a binary stream, which can be any string.

[0134] S107. Receive the SPL search statement input by the user.

[0135] In this embodiment of the application, SQL, or Structured Query Language, is a special-purpose programming language, a database query and programming language used to access data and query, update and manage relational database systems.

[0136] In this embodiment of the application, the SPL search statement includes full-text search statement, structured search statement, combined filtering statement, etc., and this embodiment of the application does not limit the scope of the search statement.

[0137] In this embodiment of the application, the syntax of the full-text search statement is value, for example: book, which means that among all the values ​​of all fields, the data whose value contains "book", "book", and "book".

[0138] In this embodiment, the structured search statement syntax is field_name opr field_value, where field_name represents the field name, which can be letters, numbers, or underscores, and the first character must be a letter; field_value represents the field value, which can be numbers, Chinese characters, strings, IP addresses, etc.; opr represents the relational connector, supporting symbols such as >, <, =, >=, <=, !=, etc. For example: field_name = book, which means querying all values ​​of the field_name field that contain the characters "book", "book", or "book". Another example: field_name = "book", which means querying all values ​​of the field_name field that contain the character "book".

[0139] In this embodiment, the syntax of the combined filtering statement is a full-text search statement / structured search statement (OPR). OPR supports AND and OR; OR means that the result set only needs to satisfy the conditions on either side of the OR operator; AND means that the result set only needs to satisfy the conditions on either side of the OR operator. For example: field = "malicious code" OR field = "information harm", or field = "malicious code" AND field = "information harm" AND 111.111.11.11, etc., where 111.111.11.11 is an IP address.

[0140] S108. Parse the SPL search statement to obtain a statement queue containing multiple search sub-statements.

[0141] As an optional implementation, the SPL search statement is parsed to obtain a statement queue including multiple search sub-statements, including:

[0142] The SPL retrieval statement is parsed using a pre-compiled abstract syntax tree to obtain an abstract syntax tree;

[0143] Traverse the entire syntax tree and identify the specific statement based on the type of the leaf node;

[0144] The identified statements are recorded in the predefined statement queue.

[0145] In the above implementation, parsing the SPL search statement aims to parse the user-input search analysis language into individual statements and identify the type of each statement. Specifically, the user-input SPL search analysis language is sent to the language processing engine, which then uses a pre-compiled abstract syntax tree to parse the statements. The language processing engine can parse SPL into an abstract syntax tree, where the leaf nodes are individual statements. The entire syntax tree is traversed, and the specific statements are identified based on the type of the leaf nodes and recorded in a predefined statement queue.

[0146] S109. Convert each retrieval sub-statement in the statement queue into an Elasticsearch query statement.

[0147] In this embodiment, the query statement can specifically be a term query statement. A term query generally expresses the smallest unit of query, searching for keywords.

[0148] As an optional implementation, each retrieval sub-statement in the statement queue is converted into an Elasticsearch query statement, including:

[0149] Iterate through each search sub-statement in the statement queue;

[0150] Each retrieval sub-statement is converted according to the query syntax of the target storage medium to obtain the target storage medium data retrieval and analysis statement corresponding to each retrieval sub-statement;

[0151] The transformed target storage medium data retrieval and analysis statements are assembled into Elasticsearch query statements according to the order of the SPL statement queue.

[0152] In the above implementation, during the conversion, for example, a structured search statement can be converted into an Elasticsearch term query statement, and a field projection statement can be converted into an Elasticsearch source query statement.

[0153] In the above implementation, the Elasticsearch query statement is the statement that ultimately retrieves and analyzes data from the target storage medium.

[0154] S110. Based on the query statement, call the Elasticsearch data operation interface to perform the query and obtain the Elasticsearch query results.

[0155] In this embodiment of the application, when performing an Elasticsearch query, based on the transformed query statement, the data operation API (i.e., data operation interface) provided by the Elasticsearch database is called to complete the query of data under the storage medium and obtain the query results.

[0156] S111. Obtain HBase query results based on Elasticsearch query results.

[0157] As an optional implementation, the HBase query results are obtained based on the Elasticsearch query results, including:

[0158] The Elasticsearch query results are parsed to obtain the parsed data;

[0159] Retrieve the value of the rowkey field from the parsed data;

[0160] Convert the value of the rowkey field into an HBase query statement;

[0161] Connect to the HBase database, perform queries on the connected HBase database using HBase query statements, and obtain the HBase query results.

[0162] In the above implementation, the Elasticsearch query results are parsed, the rowkey field value in the parsed data is converted into an HBase query statement, and finally, the raw data is obtained by connecting to HBase. This raw data is the HBase query result.

[0163] S112. Merge the Elasticsearch query results and HBase query results to obtain the SPL retrieval results.

[0164] In this embodiment of the application, the Elasticsearch query results and HBase query results are merged one by one to obtain the SPL retrieval results, and then returned.

[0165] In this embodiment of the application, through steps S107 to S112, a search statement based on a custom data retrieval language can be received to complete the data query operation. This effectively solves the problem that when R&D personnel perform interactive retrieval analysis on massive amounts of data, they need to interact with two storage systems, manually merge the query results, and learn the Elasticsearch retrieval analysis language. This effectively reduces the complexity of data retrieval analysis.

[0166] In this embodiment, the method addresses massive amounts of data simultaneously stored on different storage media. It utilizes a data processing engine to handle data writing, data synchronization between multiple storage media, data lifecycle maintenance, and unified data querying. The data processing engine, combined with SPL (Search Process Language), queries the data, avoiding direct manipulation of multi-source data by developers. This solves the problems of high development costs and complex maintenance associated with multi-source data storage, reducing the complexity of development for big data developers, improving efficiency, and enhancing quality. Furthermore, it lowers the learning curve and provides a better user experience for data analysts performing interactive data retrieval and analysis.

[0167] In this embodiment, the input query SPL statement is: DVC_ID = 123 AND time >= 2021-12-20. This example query SPL statement can be parsed. After being analyzed by the syntax analyzer inside the language processing engine, it will form two conditional statements ("DVC_ID = x123", "time >= 2021-12-20"). The two parsed clauses are then recorded sequentially in the statement queue. Finally, the parsed statement is converted into the corresponding Elasticsearch data retrieval statement. An example of the converted statement is as follows:

[0168] Specifically, "DVC_ID=123" is converted to:

[0169]

[0170] Specifically, "time>=2021-12-20" is converted to:

[0171]

[0172] Next, send the transformed Elasticsearch query from the previous step to the Elasticsearch server to retrieve the data stored in Elasticsearch. An example of the retrieved data is shown below:

[0173]

[0174] Example of a query statement in this embodiment:

[0175] get'alarm_event','1648372839000000absczdrfs'

[0176] In this embodiment of the application, the query result example corresponding to the above query statement example is as follows:

[0177]

[0178] In this embodiment, implementing this method eliminates the need for manual development and maintenance of real-time data synchronization, data query merging, and lifecycle management of multi-source historical data. It also eliminates the need for in-depth knowledge of different storage media, reducing development complexity, improving R&D efficiency, and enhancing R&D quality. Furthermore, it solves the data write consistency problem and reduces the complexity of data writing. Using a custom SPL language within the data processing engine, interactive retrieval and analysis requires interaction with two systems and the merging of query results, necessitating learning Elasticsearch's retrieval and analysis language, thus reducing the complexity of data retrieval and analysis. Finally, using the data processing engine for data lifecycle management reduces development and maintenance costs.

[0179] As can be seen, implementing the data operation method based on the data processing engine described in this embodiment can avoid researchers from directly manipulating multi-source data, thereby reducing the development complexity and maintenance cost of multi-source data storage, and improving the research and development efficiency and quality of researchers.

[0180] Example 2

[0181] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a data operating system based on a data processing engine, provided in this embodiment. For example... Figure 2 As shown, the data operating system based on the data processing engine includes:

[0182] Configuration unit 201 is used to configure startup configuration information in the data processing engine; the startup configuration information includes at least an outline file of the target topic data group in the open source stream processing platform;

[0183] Processing unit 202 is used to deserialize the outline document to obtain the target topic data set;

[0184] Create unit 203 to create a listening thread based on the target topic data group;

[0185] The generation unit 204 is used to generate HBase data writing statements based on a pre-created HBase data table template when the listening thread detects data to be written from the open-source stream processing platform.

[0186] Write unit 205 is used to write HBase data write statements to the HBase database.

[0187] As an optional implementation, the generation unit 204 is specifically used to generate a unique data identifier that matches the data to be written when the listening thread detects the data to be written from the open-source stream processing platform; and to generate an HBase data write statement based on the unique data identifier and a pre-created HBase data table template; the HBase data write statement includes the unique data identifier.

[0188] As an optional implementation, the data operating system based on the data processing engine further includes:

[0189] The startup unit 206 is used to start the HBase scanner via a MapReduce task after the step of writing the HBase data write statement to the HBase database.

[0190] The listening unit 207 is used to scan the latest data unique identifier in HBase cyclically according to a preset interval.

[0191] The judgment unit 208 is used to determine whether the latest data unique identifier is greater than the last data unique identifier; the last data unique identifier is the latest data unique identifier scanned by the scanner in the last scan.

[0192] The acquisition unit 209 is used to acquire the latest HBase data that matches the latest data unique identifier when the latest data unique identifier is greater than the last data unique identifier.

[0193] The generation unit 204 is also used to convert the latest HBase data into structured code data; and to write the code data as Elasticsearch data into Elasticsearch, and generate an Elasticsearch index based on the latest data unique identifier and a pre-created Elasticsearch index template.

[0194] As an optional implementation, the data operating system based on the data processing engine further includes:

[0195] The acquisition unit 209 is also used to retrieve Elasticsearch storage data in Elasticsearch in a loop according to a preset interval by using the data lifecycle management thread.

[0196] The judgment unit 208 is also used to determine whether the Elasticsearch stored data has expired or whether the storage space occupied by the Elasticsearch stored data is too large, based on a preset time management strategy or a preset space management strategy.

[0197] The deletion unit 210 is used to delete the Elasticsearch index of the Elasticsearch data with the smallest unique identifier matched by the daily prefix when the Elasticsearch stored data expires or the storage space occupied by the Elasticsearch stored data is too large.

[0198] As an optional implementation, the data operating system based on the data processing engine further includes:

[0199] The receiving unit 211 is used to receive the SPL search statement input by the user;

[0200] The parsing unit 212 is used to parse the SPL retrieval statement to obtain a statement queue including multiple retrieval sub-statements;

[0201] Transformation unit 213 is used to convert each retrieval sub-statement in the statement queue into an Elasticsearch query statement;

[0202] Calling unit 214 is used to call the Elasticsearch data operation interface to perform a query based on the query statement and obtain the Elasticsearch query results;

[0203] The query unit 215 is used to obtain HBase query results based on Elasticsearch query results; and to merge the Elasticsearch query results and HBase query results to obtain SPL retrieval results.

[0204] In this embodiment, the explanation of the data operating system based on the data processing engine can be referred to the description in Embodiment 1, and will not be repeated here.

[0205] As can be seen, implementing the data operating system based on the data processing engine described in this embodiment can avoid researchers from directly manipulating multi-source data, thereby reducing the development complexity and maintenance cost of multi-source data storage, and improving the research and development efficiency and quality of researchers.

[0206] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the data operation method based on a data processing engine as described in embodiment 1 of this application.

[0207] This application provides a computer-readable storage medium storing computer program instructions. When these computer program instructions are read and executed by a processor, they perform the data operation method based on a data processing engine as described in embodiment 1 of this application.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0209] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0210] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0211] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0212] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0213] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data manipulation method based on a data processing engine, characterized in that, include: Configure startup configuration information in the data processing engine; The startup configuration information includes at least an outline file of the target topic data group in the open-source stream processing platform; The outline document is deserialized to obtain the target topic data set; A listening thread is created based on the target topic data group; the target topic data group is a list of topic names. When the listening thread detects the data to be written from the open-source stream processing platform, it generates an HBase data write statement based on the pre-created HBase data table template. The data write statement in HBase is written to the HBase database.

2. The data manipulation method based on a data processing engine according to claim 1, characterized in that, The step of generating an HBase data write statement based on a pre-created HBase data table template when the listening thread detects data to be written from the open-source stream processing platform includes: When the listening thread detects the data to be written from the open-source stream processing platform, it generates a unique data identifier that matches the data to be written. Based on the unique data identifier and the pre-created HBase data table template, an HBase data write statement is generated; the HBase data write statement includes the unique data identifier.

3. The data manipulation method based on a data processing engine according to claim 1, characterized in that, After the step of writing the HBase data write statement to the HBase database, the method further includes: The HBase scanner is started via a MapReduce task; According to a preset interval, the scanner cyclically scans the latest unique identifier of the data in the HBase; Determine whether the latest data unique identifier is greater than the last data unique identifier; the last data unique identifier is the latest data unique identifier detected by the scanner in the last scan. When the latest data unique identifier is greater than the last data unique identifier, obtain the latest HBase data that matches the latest data unique identifier; Convert the latest HBase data into structured code data; The code data is written to Elasticsearch as Elasticsearch data, and an Elasticsearch index is generated based on the latest data's unique identifier and a pre-created Elasticsearch index template.

4. The data manipulation method based on a data processing engine according to claim 1, characterized in that, The method further includes: The data lifecycle management thread retrieves Elasticsearch storage data in Elasticsearch in a loop according to a preset interval. Based on a preset time management strategy or a preset space management strategy, determine whether the Elasticsearch stored data has expired or whether the storage space occupied by the Elasticsearch stored data is too large. When the Elasticsearch stored data expires or the storage space occupied by the Elasticsearch stored data becomes too large, delete the Elasticsearch index of the Elasticsearch data with the smallest unique identifier matched by the daily prefix.

5. The data manipulation method based on a data processing engine according to claim 1, characterized in that, The method further includes: Receive SPL search statements input by the user; The SPL retrieval statement is parsed to obtain a statement queue containing multiple retrieval sub-statements; Each retrieval sub-statement in the statement queue is converted into an Elasticsearch query statement; Based on the query statement, call the Elasticsearch data manipulation interface to perform the query and obtain the Elasticsearch query results; Based on the Elasticsearch query results, obtain the HBase query results; The Elasticsearch query results and the HBase query results are merged to obtain the SPL retrieval results.

6. A data operating system based on a data processing engine, characterized in that, The data operating system based on the data processing engine includes: A configuration unit is used to configure startup configuration information in the data processing engine; the startup configuration information includes at least an outline file of the target topic data group in the open-source stream processing platform. A processing unit is used to deserialize the outline document to obtain the target topic data group; A creation unit is used to create a listening thread based on the target topic data group; the target topic data group is a list of topic names. The generation unit is used to generate HBase data writing statements based on a pre-created HBase data table template when the listening thread detects the data to be written from the open-source stream processing platform. The write unit is used to write the data write statement of the HBase database.

7. The data operating system based on a data processing engine according to claim 6, characterized in that, The generation unit is specifically used to generate a unique data identifier that matches the data to be written when the listening thread detects the data to be written on the open-source stream processing platform; and to generate an HBase data write statement based on the unique data identifier and a pre-created HBase data table template; the HBase data write statement includes the unique data identifier.

8. The data operating system based on a data processing engine according to claim 6, characterized in that, The data operating system also includes: The startup unit is used to start the HBase scanner via a MapReduce task after the step of writing the data write statement of the HBase to the HBase database; The monitoring unit is used to cyclically scan the latest data unique identifier in the HBase according to a preset interval. The judgment unit is used to determine whether the latest data unique identifier is greater than the last data unique identifier; the last data unique identifier is the latest data unique identifier scanned by the scanner in the last scan. The acquisition unit is used to acquire the latest HBase data that matches the latest data unique identifier when the latest data unique identifier is greater than the last data unique identifier; The generation unit is further configured to convert the latest HBase data into structured code data; write the code data as Elasticsearch data into Elasticsearch; and generate an Elasticsearch index based on the unique identifier of the latest data and a pre-created Elasticsearch index template.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the data manipulation method based on the data processing engine as described in any one of claims 1 to 5.

10. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the data manipulation method based on a data processing engine as described in any one of claims 1 to 5.

Images