Mass data processing method, system and equipment, medium and program product
The combination of Elasticsearch, HBase, and Kafka solves the complexity issues of massive data storage and query in traditional databases, achieves efficient data storage and fast query, and improves user experience.
Patent Information
- Application Number
- CN202510789562.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-13
- Publication Date
- 2025-09-19
AI Technical Summary
Traditional databases are unable to support massive data storage, and data queries are complex and have high performance requirements, which affects user experience.
A combination of Elasticsearch, HBase, and Kafka is used. HBase stores massive data, Elasticsearch is responsible for complex queries and full-text retrieval, and Kafka implements data synchronization, achieving separation and flexible expansion of storage and query.
It provides efficient writing and long-term storage of massive data, supports millisecond-level query responses, and improves user experience.
Smart Images
Figure CN120670518A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, system, device, medium and program product for processing massive data. Background Art
[0002] In the era of data explosion, as business scale continues to expand, data storage faces the following major challenges: 1) Large data volumes make it difficult for traditional databases to support tens or even hundreds of billions of data; 2) Complex queries and diverse business needs require support for multi-condition queries, fuzzy queries, and aggregate queries; 3) High performance requirements require query response times in milliseconds, otherwise the user experience will be affected; 4) High availability requires the system to operate stably 24 / 7 to avoid single points of failure. Summary of the Invention
[0003] The technical problem to be solved by the present disclosure is to overcome the defects of traditional databases in the prior art, such as not supporting the storage of massive data and complex data queries, and to provide a method, system, device, medium and program product for processing massive data.
[0004] The present disclosure solves the above technical problems through the following technical solutions:
[0005] A first aspect of the present disclosure provides a method for processing massive data, the method being applicable to Elasticsearch (a search server), and comprising:
[0006] In response to writing massive amounts of data into Hbase (an open-source database), a Kafka (an open-source stream processing platform) request is triggered, and the massive amounts of data are pushed to Kafka.
[0007] Pulling the massive data from the Kafka, parsing the massive data, and writing the parsed massive data into Elasticsearch;
[0008] receiving a target data query request;
[0009] Obtain the target row key of the target data from Redis (remote dictionary service) according to the target data query request;
[0010] Query target data from the Hbase according to the target row key.
[0011] Preferably, the processing method further comprises:
[0012] In response to writing the parsed massive data into Elasticsearch, a prompt message indicating that the massive data synchronization is completed is output.
[0013] Preferably, the processing method further comprises:
[0014] In response to not writing the parsed massive data into Elasticsearch, the massive data is re-pulled from the Kafka via Elasticsearch.
[0015] Preferably, the processing method further comprises:
[0016] Get row keys for massive amounts of data;
[0017] The row keys of the massive data are stored in Redis.
[0018] Preferably, the processing method further comprises:
[0019] An index is created in the Elasticsearch based on the row keys of the massive data.
[0020] Preferably, the step of obtaining the target row key of the target data from the redis according to the target data query request includes:
[0021] The target row key of the target data is obtained from the redis according to the target data query request and the index.
[0022] A second aspect of the present disclosure provides a system for processing massive data, the system being applicable to Elasticsearch and comprising:
[0023] A push module is used to trigger a Kafka request in response to writing massive data into Hbase, and push the massive data to Kafka;
[0024] A writing module is used to pull the massive data from the Kafka, parse the massive data, and write the parsed massive data into Elasticsearch;
[0025] A receiving module, configured to receive a target data query request;
[0026] The first acquisition module is used to obtain the target row key of the target data from Redis according to the target data query request;
[0027] The query module is used to query the target data from the Hbase according to the target row key.
[0028] Preferably, the processing system further comprises:
[0029] The output module is used to output a prompt message indicating that the massive data synchronization is completed in response to writing the parsed massive data into Elasticsearch.
[0030] Preferably, the processing system further comprises:
[0031] The pulling module is used to re-pull the massive data from the Kafka through Elasticsearch in response to the massive data after parsing not being written into Elasticsearch.
[0032] Preferably, the processing system further comprises:
[0033] The second acquisition module is used to obtain row keys of massive data;
[0034] The storage module is used to store the row keys of the massive data in redis.
[0035] Preferably, the processing system further comprises:
[0036] A module is established, for establishing an index in the Elasticsearch according to row keys of massive data.
[0037] Preferably, the first acquisition module is used to obtain the target row key of the target data from the redis according to the target data query request and the index.
[0038] A third aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein when the processor executes the computer program, the method for processing massive data described in the first aspect is implemented.
[0039] A fourth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for processing massive data described in the first aspect is implemented.
[0040] A fifth aspect of the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for processing massive data as described in the first aspect.
[0041] On the basis of conforming to the common sense in this field, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.
[0042] The positive progress of this disclosure is:
[0043] This paper adopts the combination of Elasticsearch, HBase and Kafka to achieve complementary advantages, separation of storage and query, and flexible expansion. Specifically, HBase is used to store massive data, ensuring efficient writing and long-term preservation of massive data. Elasticsearch is responsible for complex queries and full-text retrieval, providing millisecond-level query response. Kafka is used to achieve data synchronization between Elasticsearch and HBase, improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is a flowchart of a method for processing massive data provided in Example 1 of the present disclosure.
[0045] Figure 2 A schematic diagram of the modules of a system for processing massive amounts of data provided in Example 2 of the present disclosure.
[0046] Figure 3 This is a structural diagram of an electronic device for implementing a method for processing massive data according to embodiment 3 of the present disclosure. DETAILED DESCRIPTION
[0047] The present disclosure is further illustrated below by way of examples, but the present disclosure is not limited to the scope of the examples.
[0048] In the embodiments of the present disclosure, prefixes such as "first" and "second" are used only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. In the embodiments of the present disclosure, the use of prefixes such as ordinal numbers to distinguish description objects does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitations should be constituted due to the use of such prefixes. In addition, in the description of this embodiment, unless otherwise specified, the meaning of "plurality" is two or more.
[0049] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0050] Example 1
[0051] Figure 1 This is a flowchart of a method for processing massive data provided in Example 1 of the present disclosure. The processing method is applicable to Elasticsearch, such as Figure 1 As shown, the processing method includes:
[0052] S1, in response to writing massive data into Hbase, triggers a Kafka request and pushes the massive data to Kafka;
[0053] In this embodiment, after massive data (for example, tens of billions of data) is written into HBase, a Kafka request is triggered (for example, a Kafka message producer is triggered), and the massive data change event is pushed to Kafka.
[0054] S2: Pull massive data from Kafka, parse it, and write the parsed data into Elasticsearch.
[0055] In this embodiment, the Elasticsearch consumer pulls massive data from Kafka, parses it, and writes it to Elasticsearch.
[0056] S3. Receive target data query request;
[0057] S4. Obtain the target row key of the target data from Redis according to the target data query request;
[0058] S5. Query the target data from HBase based on the target row key.
[0059] This embodiment uses a combination of Elasticsearch, HBase, and Kafka to achieve complementary advantages, separation of storage and query, and flexible expansion. Specifically, HBase is used to store massive data, ensuring efficient writing and long-term preservation of massive data. Elasticsearch is responsible for complex queries and full-text retrieval, providing millisecond-level query responses. Kafka is used to achieve data synchronization between Elasticsearch and HBase, improving user experience.
[0060] In an optional embodiment, the processing method further includes:
[0061] In response to writing the parsed massive data into Elasticsearch, a prompt message indicating that the massive data synchronization is completed is output.
[0062] In this embodiment, if the parsed massive data is successfully written into Elasticsearch, it is determined that the massive data synchronization is completed, and a prompt message indicating that the massive data synchronization is completed is output.
[0063] In an optional embodiment, the processing method further includes:
[0064] In response to the massive amount of parsed data not being written into Elasticsearch, the massive amount of data is re-pulled from Kafka via Elasticsearch.
[0065] In an optional embodiment, the processing method further includes:
[0066] Get row keys for massive amounts of data;
[0067] Store the row keys of massive data in Redis.
[0068] In an optional embodiment, the processing method further includes:
[0069] Create indexes in Elasticsearch based on the row keys of massive data.
[0070] In an optional embodiment, S4 includes:
[0071] The target row key of the target data is obtained from the redis according to the target data query request and the index.
[0072] During the specific implementation process, the massive data writing process is as follows: after the massive data is written to HBase, a Kafka request is triggered to push the massive data to Kafka; the Elasticsearch consumer pulls the massive data from Kafka, parses the massive data, and writes it to Elasticsearch; if writing to Elasticsearch fails, the massive data will be re-queued (for example, the massive data will be re-pulled from Kafka through Elasticsearch) to ensure eventual consistency; if the parsed massive data is successfully written to Elasticsearch, the massive data synchronization is determined to be complete, and a prompt message indicating the completion of the massive data synchronization is output.
[0073] The process for reading massive amounts of data is as follows: The HBase rowkey is set to the Elasticsearch document ID. During a search, the corresponding document is first retrieved from Elasticsearch based on the business conditions. This is to obtain the document ID (i.e., the rowkey). The data is then extracted from HBase. Specifically, this involves storing massive amounts of data in HBase and simultaneously storing the rowkeys of the massive data in Redis.
[0074] Obtain the rowkey of massive data from Redis, query data in HBase based on the rowkey, and create an index in Elasticsearch; after the business side receives the target data query request (for example, the user sends a target data query request to Elasticsearch), Elasticsearch returns the rowkey that meets the conditions (for example, the target rowkey); query the target data from HBase based on the rowkey of the target data; HBase returns the target data corresponding to the rowkey.
[0075] In this embodiment, in the scenario of data storage at the billion level, HBase and Elasticsearch have unique advantages in data storage and query, and can complement each other perfectly to solve core problems such as large data volume, complex query, and high performance. The advantages of HBase are as follows: 1) Massive storage: HBase is a distributed columnar storage system built on HDFS (a highly fault-tolerant system) with extremely strong scalability. It can easily support the storage of tens of billions or hundreds of billions of data. Through Region automatic sharding and dynamic expansion mechanism, it can linearly expand storage capacity as the amount of data grows; 2) High throughput: HBase has excellent write performance and adopts LSM (log-structured merge tree) tree storage structure. When writing data, it is first written to the memory, and then flushed to the disk in batches when the threshold is reached, which greatly improves the write throughput; 3) Efficient row key query: supports fast query based on row keys. The advantages of Elasticsearch are as follows: 1) Full-text search: supports advanced query functions such as word segmentation, fuzzy query, and multi-condition combination query; 2) Aggregation analysis: provides powerful aggregation functions, supports complex analysis operations, and can directly extract statistical results from indexed data; 3) High performance: through inverted indexing and distributed architecture, it can quickly locate target data and return results.
[0076] This embodiment implements massive data storage based on Elasticsearch and HBase, achieving complementary advantages, separation of storage and query, and flexible expansion. HBase achieves linear expansion of storage capacity by adding RegionServers, and Elasticsearch achieves horizontal expansion of query performance by adding nodes. Specifically, Elasticsearch supports complex query capabilities, is responsible for complex queries and full-text retrieval, and provides millisecond-level query responses. It can quickly locate target data and return results through inverted indexes and distributed architecture; HBase supports massive storage, is responsible for massive data storage, supports high-throughput writing and efficient row key queries, and ensures efficient data writing and long-term preservation. Kafka supports high-concurrency, low-latency message processing, and uses Kafka to implement asynchronous data synchronization between Elasticsearch and HBase.
[0077] Example 2
[0078] Corresponding to the aforementioned embodiment of a method for processing massive data, the present disclosure also provides an embodiment of a system for processing massive data.
[0079] Figure 2 This is a module diagram of a massive data processing system provided in Example 1 of the present disclosure. The processing system is applicable to Elasticsearch, such as Figure 2 As shown, the processing system includes:
[0080] Push module 21, for triggering a Kafka request in response to writing massive data into Hbase, and pushing the massive data to Kafka;
[0081] In this embodiment, after massive data (for example, tens of billions of data) is written into HBase, a Kafka request is triggered (for example, a Kafka message producer is triggered), and the massive data change event is pushed to Kafka.
[0082] The writing module 22 is used to pull massive data from Kafka, parse the massive data, and write the parsed massive data into Elasticsearch;
[0083] In this embodiment, the Elasticsearch consumer pulls massive data from Kafka, parses it, and writes it to Elasticsearch.
[0084] Receiving module 23, used for receiving target data query request;
[0085] The first acquisition module 24 is used to obtain the target row key of the target data from Redis according to the target data query request;
[0086] The query module 25 is used to query the target data from Hbase according to the target row key.
[0087] This embodiment uses a combination of Elasticsearch, HBase, and Kafka to achieve complementary advantages, separation of storage and query, and flexible expansion. Specifically, HBase is used to store massive data, ensuring efficient writing and long-term preservation of massive data. Elasticsearch is responsible for complex queries and full-text retrieval, providing millisecond-level query responses. Kafka is used to achieve data synchronization between Elasticsearch and HBase, improving user experience.
[0088] In an optional embodiment, the processing system further includes:
[0089] The output module is used to output a prompt message indicating that the massive data synchronization is completed in response to writing the parsed massive data into Elasticsearch.
[0090] In this embodiment, if the parsed massive data is successfully written into Elasticsearch, it is determined that the massive data synchronization is completed, and a prompt message indicating that the massive data synchronization is completed is output.
[0091] In an optional embodiment, the processing system further includes:
[0092] The pulling module is used to re-pull the massive data from Kafka through Elasticsearch in response to the massive data after parsing not being written into Elasticsearch.
[0093] In an optional embodiment, the processing system further includes:
[0094] The second acquisition module is used to obtain row keys of massive data;
[0095] The storage module is used to store the row keys of massive data in Redis.
[0096] In an optional embodiment, the processing system further includes:
[0097] Build a module for indexing massive amounts of data in Elasticsearch based on row keys.
[0098] In an optional embodiment, the first acquisition module is used to obtain the target row key of the target data from Redis according to the target data query request and the index.
[0099] During the specific implementation process, the massive data writing process is as follows: after the massive data is written to HBase, a Kafka request is triggered to push the massive data to Kafka; the Elasticsearch consumer pulls the massive data from Kafka, parses the massive data, and writes it to Elasticsearch; if writing to Elasticsearch fails, the massive data will be re-queued (for example, the massive data will be re-pulled from Kafka through Elasticsearch) to ensure eventual consistency; if the parsed massive data is successfully written to Elasticsearch, the massive data synchronization is determined to be complete, and a prompt message indicating the completion of the massive data synchronization is output.
[0100] The process for reading massive data is as follows: The HBase rowkey is set to the Elasticsearch document ID. During a search, the corresponding document is first retrieved from Elasticsearch based on the business conditions to obtain the document ID. After obtaining the rowkey, the data is then extracted from HBase. Specifically, the massive data is stored in HBase and the rowkey of the massive data is stored in Redis.
[0101] Obtain the rowkey of massive data from Redis, query data in HBase based on the rowkey, and create an index in Elasticsearch; after the business side receives the target data query request (for example, the user sends a target data query request to Elasticsearch), Elasticsearch returns the rowkey that meets the conditions (for example, the target rowkey); query the target data from HBase based on the rowkey of the target data; HBase returns the target data corresponding to the rowkey.
[0102] In this embodiment, in the scenario of storing exabytes of data, HBase and Elasticsearch offer unique advantages in data storage and querying. They complement each other perfectly, solving core problems such as large data volumes, complex queries, and high performance. HBase's advantages are as follows: 1) Massive Storage: HBase is a distributed columnar storage system built on HDFS, offering strong scalability and can easily support the storage of exabytes and even hundreds of exabytes of data. Through automatic region sharding and dynamic capacity expansion, storage capacity can be linearly expanded as data volume grows. 2) High Throughput: HBase offers excellent write performance, using an LSM tree storage structure. Data is first written to memory and then flushed to disk in batches when a threshold is reached, significantly improving write throughput. 3) Efficient Row Key Query: Supports fast row key-based queries. Elasticsearch's advantages are as follows: 1) Full-text Search: Supports advanced query functions such as word segmentation, fuzzy queries, and multi-condition combination queries. 2) Aggregation Analysis: Provides powerful aggregation capabilities, supports complex analytical operations, and can directly extract statistical results from indexed data. 3) High Performance: Through its inverted index and distributed architecture, HBase can quickly locate target data and return results.
[0103] This embodiment implements massive data storage based on Elasticsearch and HBase, achieving complementary advantages, separation of storage and query, and flexible expansion. HBase achieves linear expansion of storage capacity by adding RegionServers, and Elasticsearch achieves horizontal expansion of query performance by adding nodes. Specifically, Elasticsearch supports complex query capabilities, is responsible for complex queries and full-text retrieval, and provides millisecond-level query responses. It can quickly locate target data and return results through inverted indexes and distributed architecture; HBase supports massive storage, is responsible for massive data storage, supports high-throughput writing and efficient row key queries, and ensures efficient data writing and long-term preservation. Kafka supports high-concurrency, low-latency message processing, and uses Kafka to implement asynchronous data synchronization between Elasticsearch and HBase.
[0104] Since the system embodiments generally correspond to the method embodiments, reference will be made to the description of the method embodiments for relevant details. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components of the units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the disclosed solution.
[0105] Example 3
[0106] Figure 3 This is a structural diagram of an electronic device shown in Example 3 of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the method for processing massive data described in any of the above embodiments. Figure 3 The electronic device 90 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.
[0107] like Figure 3 As shown, the electronic device 90 may be a general-purpose computing device, such as a server device. Components of the electronic device 90 may include, but are not limited to, the at least one processor 91, the at least one memory 92, and a bus 93 connecting different system components (including the memory 92 and the processor 91).
[0108] The bus 93 includes a data bus, an address bus, and a control bus.
[0109] The memory 92 may include a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read-only memory (ROM) 923 .
[0110] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) of program modules 924, such program modules 924 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.
[0111] The processor 91 executes various functional applications and data processing by running the computer programs stored in the memory 92, such as the method for processing massive data provided in any of the above embodiments.
[0112] The electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.). Such communication can be performed through an input / output (I / O) interface 95. In addition, the electronic device 90 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 96. Figure 3 As shown, the network adapter 96 communicates with other modules of the electronic device 90 via the bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.
[0113] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.
[0114] Example 4
[0115] Embodiment 4 of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for processing massive data provided by any of the above embodiments is implemented.
[0116] The readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0117] Example 5
[0118] Embodiment 5 of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for processing massive amounts of data.
[0119] The program code for executing the computer program product of the present disclosure may be written in any combination of one or more programming languages, and the program code may be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on the remote device.
[0120] While specific embodiments of the present disclosure have been described above, those skilled in the art will appreciate that these are merely illustrative and that the scope of protection of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, and such changes and modifications are intended to fall within the scope of protection of the present disclosure.
Claims
1. A method for processing massive data, characterized in that: The processing method is applicable to Elasticsearch and includes: In response to writing massive data into Hbase, a Kafka request is triggered, and the massive data is pushed to Kafka; Pulling the massive data from the Kafka, parsing the massive data, and writing the parsed massive data into Elasticsearch; receiving a target data query request; Obtain the target row key of the target data from Redis according to the target data query request; Query target data from the Hbase according to the target row key.
2. The method for processing massive data according to claim 1, wherein: The processing method further comprises: In response to writing the parsed massive data into Elasticsearch, a prompt message indicating that the massive data synchronization is completed is output.
3. The method for processing massive data according to claim 1, wherein: The processing method further comprises: In response to not writing the parsed massive data into Elasticsearch, the massive data is re-pulled from the Kafka via Elasticsearch.
4. The method for processing massive data according to claim 1, wherein: The processing method further comprises: Get row keys for massive amounts of data; The row keys of the massive data are stored in Redis.
5. The method for processing massive data according to claim 1, wherein: The processing method further comprises: An index is created in the Elasticsearch based on the row keys of the massive data.
6. The method for processing massive data according to claim 5, wherein: The step of obtaining the target row key of the target data from the Redis according to the target data query request includes: The target row key of the target data is obtained from the redis according to the target data query request and the index.
7. A massive data processing system, characterized in that: The processing system is applicable to Elasticsearch, and includes: A push module is used to trigger a Kafka request in response to writing massive data into Hbase, and push the massive data to Kafka; A writing module is used to pull the massive data from the Kafka, parse the massive data, and write the parsed massive data into Elasticsearch; A receiving module, configured to receive a target data query request; The first acquisition module is used to obtain the target row key of the target data from Redis according to the target data query request; The query module is used to query the target data from the Hbase according to the target row key.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and configured to run on the processor, wherein: When the processor executes the computer program, the method for processing massive data according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for processing massive data according to any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for processing massive data according to any one of claims 1 to 6 is implemented.