Method and equipment for accelerating data parallel query based on high-frequency data processing

Abstract

The invention discloses a method and equipment for accelerating data parallel query based on high-frequency data processing, and the method comprises the steps that a first main node receives a data query request sent by a user, and the query request carries a query condition; the first main node generates an execution plan according to the data query request, and queries each first type of data node according to the execution plan; if a matching data block matched with the query condition exists in each first type of data node, the first main node returns result data to the user, and the result data is determined according to a merging result of each matching data block, wherein the high-frequency data is data of which the access frequency is greater than a preset frequency threshold in the first main node, so that the big data parallel processing capability and the query efficiency are improved.

Description

technical field

[0001] The present application relates to the technical field of big data processing, and more specifically, to a method and device for accelerating data parallel query based on high-frequency data processing. Background technique

[0002] Large-scale static data refers to a collection of data with a certain amount of data, which can provide support for accurate decision-making. It is the product of the rise and popularization of the Internet and terminal smart devices. There are also different levels of data volume, such as TB level, PB level or ZB level. . In the era of big data, its data volume is still increasing rapidly. In order to meet this large-scale data storage and processing requirements, distributed systems are widely used in the industry at present, and data is stored in multiple independent data nodes (server devices). At the same time, on this basis, the introduction of full-memory computing technology realizes that the memory can process da...

Examples

Embodiment Construction

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

[0061] As mentioned in the background art, in the prior art, data parallel processing capability and query efficiency are reduced due to data growth.

[0062] The embodiment of the present invention proposes a method for accelerating data parallel query based on high-frequency data processing. The first master node receives the data query request sent by the user, and the query request carries query conditions; the first master node according to the The data query request generates an execu...

technical field

[0001] The present application relates to the technical field of big data processing, and more specifically, to a method and device for accelerating data parallel query based on high-frequency data processing. Background technique

[0002] Large-scale static data refers to a collection of data with a certain amount of data, which can provide support for accurate decision-making. It is the product of the rise and popularization of the Internet and terminal smart devices. There are also different levels of data volume, such as TB level, PB level or ZB level. . In the era of big data, its data volume is still increasing rapidly. In order to meet this large-scale data storage and processing requirements, distributed systems are widely used in the industry at present, and data is stored in multiple independent data nodes (server devices). At the same time, on this basis, the introduction of full-memory computing technology realizes that the memory can process da...

Examples

Embodiment Construction

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

[0061] As mentioned in the background art, in the prior art, data parallel processing capability and query efficiency are reduced due to data growth.

[0062] The embodiment of the present invention proposes a method for accelerating data parallel query based on high-frequency data processing. The first master node receives the data query request sent by the user, and the query request carries query conditions; the first master node according to the The data query request generates an execu...

Images