Method and apparatus for processing large-scale data based on stream computing

Abstract

The application provides a large-scale data processing method and device based on stream computing, and the method comprises the following steps: establishing an HTTP service between a data provider and a data processor, wherein the HTTP service converts data in an HTTP protocol message body into a binary data packet; sending the binary data packet to a Kafka message queue; creating a deserializer in a process of constructing a consumer program of a Flink computing framework, so as to construct a new Flink computing framework; and performing data processing on the Kafka message queue based on the new Flink computing framework, so as to obtain a data processing result. The application reduces the data access difficulty of a stream data processing system and improves the throughput of the entire data processing system.

Description

Technical Field

[0001] This invention belongs to the field of big data technology, specifically proposing a large-scale data processing method and apparatus based on stream computing. Background Technology

[0002] When processing internet-scale data and similar business processes, stream computing is typically used to improve real-time data processing capabilities and efficiency. During data or log collection, a message queue is needed as a data relay and distribution component, and to provide temporary data storage functionality.

[0003] Kafka is a distributed message queue designed for big data processing, characterized by low latency, high throughput, and high reliability. To work with a message queue, a distributed stream processing framework is needed to process the data. Current technology typically uses Kafka as a message queue by serializing individual data entries into JSON format and sending them to the broker; the computing framework can easily handle this data format. However, this approach has several problems:

[0004] 1. The protocol for accessing Kafka is relatively complex, requiring the integration of client libraries and the corresponding API versions of applications, and it is not friendly to data writing programs implemented in various programming languages.

[0005] 2. JSON format is simple to implement, but lacks rich type support, and its serialization and deserialization efficiency is not high.

[0006] 3. When writing a single piece of data to a message, the message size is usually very small, which affects the throughput of the queue. Summary of the Invention

[0007] To address the aforementioned issues, this invention discloses a large-scale data processing method and apparatus based on stream computing. The data provider connects to an HTTPServer and sends data. After processing, the HTTPServer writes the data to a Kafka message queue. The Flink processing engine subscribes to the messages and executes computation tasks.

[0008] The technical content of this invention includes:

[0009] A large-scale data processing method based on stream computing, comprising the following steps:

[0010] An HTTP service is established between the data provider and the data processor, wherein the HTTP service converts the data in the HTTP protocol message body into binary data packets;

[0011] Send binary data packets to the Kafka message queue;

[0012] In the process of constructing the consumer program of the Flink computing framework, a deserializer is created to build the new Flink computing framework;

[0013] Based on the new Flink computing framework, data processing is performed on the Kafka message queue to obtain the data processing results.

[0014] Furthermore, the data format in the HTTP protocol message body includes: CSV format and / or JSON format.

[0015] Furthermore, the HTTP service converts the data in the HTTP protocol message body into binary data packets, including:

[0016] Before data processing, the description information of all basic type fields of the data is preloaded into the SchemaRegistry component;

[0017] During data processing, the HTTP service obtains the AvroSchema from the SchemaRegistry component based on the message queue Topic, and then parses the data in the HTTP protocol message body according to the field order and field type to obtain the value of each field.

[0018] According to Avro's serialization method, the value of each basic field is converted into binary data to obtain the binary data packet.

[0019] Furthermore, the basic types include: Boolean, Int, Long, Float, Double, String, and Null.

[0020] Furthermore, the data in the HTTP protocol message body is parsed according to the field order and field type to obtain the value of each basic field, including:

[0021] For a basic field of type Boolean, the value of the basic field is obtained by determining whether the basic field is false or true;

[0022] For basic fields of type Int or Long, the value of the basic field is obtained by following a variable-length, zig-zag encoding pattern;

[0023] For basic fields of type Float, the value of the basic field is obtained by converting it to 4-byte encoding using Java's floatToIntBytes method and outputting it in little-endian byte order.

[0024] For a basic field of type Double, the value of the basic field is obtained by converting it to an 8-byte encoding using Java's doubleToLongBytes method and outputting it in little-endian byte order.

[0025] For a basic field of type String, the value of the basic field is obtained by describing the number of bytes using the encoding of type Long, followed by UTF-8 encoded character data.

[0026] For null type basic fields, the value of the basic field is not output.

[0027] Furthermore, sending the binary data packet to the Kafka message queue includes:

[0028] The binary data packet is sent to the Kafka message queue by writing multiple lines of data into a single Kafka message.

[0029] Furthermore, the type attribute of the deserializer includes: DeserializationSchema.

[0030] Furthermore, the data processing of the Kafka message queue based on the new Flink computing framework to obtain data processing results includes:

[0031] Construct a consumer program based on the Kafka message queue;

[0032] The consumer program retrieves the schema corresponding to the message in the message queue from the schemaRegistry;

[0033] Based on the schema, the data in the Kafka message queue is parsed field by field.

[0034] The parsed data is used as the result of data processing.

[0035] A storage medium storing a computer program, wherein the computer program is configured to execute any of the methods described above when a processor is run.

[0036] An electronic device is characterized by comprising a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform any of the methods described above.

[0037] The method proposed in this invention has the following advantages and effects:

[0038] 1. Reduce the difficulty of data access for streaming data processing systems.

[0039] 2. Avro has rich type support and is a binary protocol, so the serialized messages take up less space and save bandwidth.

[0040] 3. The method of merging and sending multiple lines of data can improve the throughput of the entire data processing system. Attached Figure Description

[0041] Figure 1 Overall structural diagram of the present invention.

[0042] Figure 2 A flowchart of Flink's message queue processing. Detailed Implementation

[0043] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely specific embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] The overall structure of the present invention is as follows Figure 1 As shown, the data provider connects to the HTTPServer and sends data. After processing, the HTTPServer writes the data to the Kafka message queue. The Flink processing engine subscribes to the messages and executes computation tasks, including the following steps:

[0045] Step 1: Establish a service using the HTTP protocol and obtain data packets based on the HTTP protocol message body.

[0046] HTTP is a commonly used network protocol, and most common programming languages ​​have corresponding implementations.

[0047] In one example, after receiving data in formats such as CSV and JSON, the HTTPServer parses the data and performs Avro serialization to form individual binary data packets.

[0048] Typically, CSV and JSON data consist of multiple basic data type fields. During parsing, the basic information of these fields is described using AvroSchema. The message queue topic and AvroSchema are defined by the data provider and pre-written into the SchemaRegistry before data loading, where they are managed uniformly. After obtaining the AvroSchema from the SchemaRegistry component based on the message queue topic, the HTTPServer program parses the received CSV or JSON data according to the field order and type. After obtaining the value of each field, it converts the value into binary data using Avro's serialization method. Generally, CSV or JSON data contains fields of basic data types, supporting Boolean, Int, Long, Float, Double, String, and Null. Boolean data uses a single byte of 0 or 1, where 0 represents false and 1 represents true. Int and Long types follow a variable-length, zig-zag encoding pattern. Float data is converted to 4 bytes using Java's floatToIntBytes method and output in little-endian byte order. Double data is converted to 8 bytes using Java's doubleToLongBytes method, also output in little-endian byte order. String data consists of a Long type encoding describing the number of bytes, followed by UTF-8 encoded character data. Null data does not output any bytes. A single data entry composed of multiple basic data type fields is called a Record in Avro's type system. When serializing multiple rows of data, the Array type is selected from the composite types. A Long type binary data entry representing the number of array elements is written first, followed by the elements themselves.

[0049] Step 2: Encapsulate the data packet into a message and send it to the Kafka message queue.

[0050] In one example, when data packets are sent to the Kafka message queue, multiple lines of data are written into a single message. After merging the multiple lines of data, the message body in the Kafka message queue becomes larger. The network traffic of the Kafka node is positively correlated with the message body size. The relevant test results are shown in Table 1:

[0051] 100 bytes 821，557 78.3MB / s 1K 532，183 519.7MB / s 10K 109，922 1073.4MB / s

[0052] Table 1

[0053] Furthermore, in conjunction with data compression, merging data can significantly improve the resource utilization of nodes.

[0054] Step 3: Modify the Flink data processing flow to adapt to the message encapsulation method.

[0055] Flink is a popular computing framework that provides data processing interfaces, computing resource management, and other functions, while supporting stateful stream computing and ensuring fault tolerance through checkpoints and savepoints.

[0056] In Flink's data processing, the data goes through various parts, including the Source, operators, and Sink. The Source part contains the consumer program for the Kafka node. Modifying Flink's data processing flow involves creating a deserializer within the consumer program construction process to parse messages. The deserializer has a property of type DeserializationSchema, which defines how to parse the binary data in the Kafka message body. Through this interface, a batch data parsing function similar to Avro can be implemented.

[0057] After modifying Flink's data processing, such as Figure 2 As shown, the consumer program retrieves the schema corresponding to the message in the message queue from the SchemaRegistry and begins the parsing process. Based on the previous data serialization method, it parses the data field by field according to the retrieved schema. If there are still bytes remaining in the data packet after parsing a row, the decoder will continue working until the entire data packet is read. The parsed data is returned as an iterator, so it is retrieved from the Source as a single row of data.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A large-scale data processing method based on stream computing, comprising the following steps: An HTTP service is established between the data provider and the data processor, wherein the HTTP service converts the data in the HTTP protocol message body into binary data packets; The binary data packet is sent to the Kafka message queue by writing multiple lines of data into a single Kafka message. In the process of constructing the consumer program of the Flink computing framework, a deserializer is created to build the new Flink computing framework; Based on the new Flink computing framework, data processing is performed on the Kafka message queue to obtain data processing results; The HTTP service converts data in the HTTP protocol message body into binary data packets, including: Before data processing, the description information of all basic type fields of the data is preloaded into the SchemaRegistry component; During data processing, the HTTP service obtains the AvroSchema from the SchemaRegistry component based on the message queue Topic, and then parses the data in the HTTP protocol message body according to the field order and field type to obtain the value of each field. According to Avro's serialization method, the value of each basic field is converted into binary data to obtain the binary data packet.

2. The method as described in claim 1, characterized in that, The data formats in the HTTP protocol message body include CSV and / or JSON formats.

3. The method as described in claim 1, characterized in that, The basic types include: Boolean, Int, Long, Float, Double, String, and Null.

4. The method as described in claim 3, characterized in that, The process involves parsing the data in the HTTP protocol message body according to the field order and field type to obtain the value of each basic field, including: For a basic field of type Boolean, the value of the basic field is obtained by determining whether the basic field is false or true; For basic fields of type Int or Long, the value of the basic field is obtained by following a variable-length, zig-zag encoding pattern; For basic fields of type Float, the value of the basic field is obtained by converting it to 4-byte encoding using Java's floatToIntBytes method and outputting it in little-endian byte order. For a basic field of type Double, the value of the basic field is obtained by converting it to an 8-byte encoding using Java's doubleToLongBytes method and outputting it in little-endian byte order. For a basic field of type String, the value of the basic field is obtained by describing the number of bytes using the encoding of type Long, followed by UTF-8 encoded character data. For null type basic fields, the value of the basic field is not output.

5. The method as described in claim 1, characterized in that, For arrays in complex types containing multiple basic types, the conversion according to Avro's serialization method also includes: Add a Long type value before the data content to describe the number of rows in the record.

6. The method as described in claim 1, characterized in that, The type attribute of the deserializer includes: DeserializationSchema.

7. The method as described in claim 1, characterized in that, The process of processing data on the Kafka message queue based on the new Flink computing framework to obtain data processing results includes: Construct a consumer program based on the Kafka message queue; The consumer program retrieves the schema corresponding to the message in the message queue from the schemaRegistry; Based on the schema, the data in the Kafka message queue is parsed field by field. The parsed data is used as the result of data processing.

8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method as described in any one of claims 1-7.

Images