Data processing method and device based on Flink engine, equipment and medium
By integrating the high-performance multilingual serialization framework Fury in the Flink engine, and using the zero-copy serialization principle and string compression algorithm, the performance bottleneck of the Flink engine in high-concurrency big data scenarios is solved, achieving more efficient and reliable data processing.
Patent Information
- Application Number
- CN202411745601.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-07
AI Technical Summary
In the high concurrency and large data volume scenarios, the Flink engine cannot fully utilize the central processor resources due to the insufficient performance of the Kryo serialization framework and increases the memory pressure of the system.
Using the data processing method based on the Flink engine, by creating thread-safe instances, the preset multilingual serialization framework Fury is integrated into the data transmission layer and state management components of the Flink engine, and the preset string compression algorithm and zero-copy serialization principle are used to serialize and deserialize operations during the data processing.
It improves the efficiency, reliability and applicability of data processing, reduces serialization and deserialization time, reduces memory consumption and data transmission delay, and improves system stability and throughput.
Smart Images
Figure CN119576880A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data processing technology, and in particular to a data processing method, device, equipment and medium based on a Flink engine. Background Art
[0002] Flink (Apache Flink, a streaming data engine) can process unbounded or bounded data streams in real time. In the Flink architecture, data is input in the form of streams, processed by a series of operators, and finally output results or stored in external systems. However, the performance of Flink is largely limited by the performance of data serialization.
[0003] Currently, Flink uses the Kryo serialization framework as a general type serialization mechanism by default. However, in scenarios with high concurrency and large amounts of data, due to the insufficient performance of Kryo, the CPU resources cannot be fully utilized. In addition, Kryo memory allocation and garbage collection may increase the memory pressure of the system, especially in long-running stream processing jobs. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a data processing method, device, equipment and medium based on the Flink engine, which can effectively improve the efficiency, reliability and applicability of data processing. The specific scheme is as follows:
[0005] In a first aspect, the present application provides a data processing method based on the Flink engine, including:
[0006] Create thread-safe instances based on the preset multi-language serialization framework, and register the object to be serialized to the obtained target instance;
[0007] Based on a preset integration strategy, the target instance is integrated into the data transmission layer and the state management component in the Flink engine to obtain a corresponding integrated computing engine;
[0008] In the process of using the integrated post-computing engine to process data, serialization operations and / or deserialization operations are triggered through a preset string compression algorithm, a zero-copy serialization principle, and the target instance to complete the corresponding data transmission operations between engine nodes and the state storage operations of the state backend of the Flink engine.
[0009] Optionally, the thread-safe instance creation based on a preset multi-language serialization framework and registering the object to be serialized to the obtained target instance includes:
[0010] Create a global thread-safe Fury instance to get the target instance;
[0011] Register the object to be serialized with the target instance.
[0012] Optionally, the step of integrating the target instance into a data transmission layer and a state management component in a Flink engine based on a preset integration strategy to obtain a corresponding integrated computing engine includes:
[0013] The general serializer in the Flink engine is closed, and the target instance is integrated into the data transmission layer, state management component, and fault recovery mechanism in the Flink engine to obtain a corresponding integrated computing engine.
[0014] Optionally, the triggering of a serialization operation and / or a deserialization operation by a preset string compression algorithm, a zero-copy serialization principle, and the target instance to complete the corresponding data transmission operation between engine nodes includes:
[0015] Based on a preset string compression algorithm, a preset digital compression algorithm, a zero-copy serialization principle and the target instance, the data to be transmitted is compressed, and the object to be serialized is automatically called to write the compressed data to be transmitted, so as to complete the data transmission operation between the corresponding engine nodes.
[0016] Optionally, compressing the data to be transmitted based on a preset string compression algorithm, a preset digital compression algorithm, a zero-copy serialization principle, and the target instance includes:
[0017] The meta-string encoding strategy, the zigzag encoding algorithm and the target instance are used to perform corresponding string compression and / or digital compression on the data to be transmitted, so as to obtain the corresponding compressed data to be transmitted.
[0018] Optionally, a serialization operation and / or a deserialization operation is triggered by a preset string compression algorithm, a zero-copy serialization principle, and the target instance to complete a state storage operation of a state backend of a corresponding Flink engine, including:
[0019] In the process of storing the operator state data of the integrated computing engine to the persistent storage system, the operator state data is compressed based on a preset string compression algorithm, a preset digital compression algorithm, a zero-copy serialization principle and the target instance, and the object to be serialized is automatically called to write data to the obtained compressed state data to complete the corresponding serialization operation.
[0020] Optionally, the method further includes:
[0021] When the fault recovery mechanism of the integrated computing engine is triggered, the object to be serialized is automatically called based on the zero-copy serialization principle and the target instance to read the compressed state data stored in the persistent storage system to complete the corresponding deserialization operation and obtain the corresponding compressed state data;
[0022] The compressed state data is decompressed using the preset string compression algorithm and the preset digital compression algorithm to obtain the corresponding operator state data, and trigger a corresponding operator state recovery operation.
[0023] In a second aspect, the present application provides a data processing device based on the Flink engine, comprising:
[0024] An instance creation module is used to create thread-safe instances based on a preset multi-language serialization framework and register the object to be serialized to the obtained target instance;
[0025] An instance integration module is used to integrate the target instance into the data transmission layer and state management component in the Flink engine based on a preset integration strategy to obtain a corresponding integrated computing engine;
[0026] The data processing module is used to trigger serialization and / or deserialization operations through a preset string compression algorithm, a zero-copy serialization principle, and the target instance during data processing using the integrated post-computing engine, so as to complete the corresponding data transmission operations between engine nodes and the state storage operations of the state backend of the Flink engine.
[0027] In a third aspect, the present application provides an electronic device, including:
[0028] Memory, used to store computer programs;
[0029] The processor is used to execute the computer program to implement the steps of the aforementioned data processing method based on the Flink engine.
[0030] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of the aforementioned data processing method based on the Flink engine.
[0031] It can be seen that in this application, thread-safe instance creation is performed based on the preset multi-language serialization framework, and the object to be serialized is registered in the obtained target instance; based on the preset integration strategy, the target instance is integrated into the data transmission layer and state management component in the Flink engine to obtain the corresponding integrated computing engine; in the process of using the integrated computing engine for data processing, the preset string compression algorithm, the zero-copy serialization principle and the target instance trigger serialization and / or deserialization operations to complete the corresponding data transmission operations between engine nodes and the state storage operations of the state backend of the Flink engine. That is, in this embodiment, by creating a thread-safe instance, the preset multi-language serialization framework is integrated into the data transmission layer and state management components in the Flink engine, and then the obtained integrated computing engine is used for data processing, and when performing data transmission operations between engine nodes and state storage operations of the engine state backend, the created target instance is used to trigger serialization or deserialization to complete data processing. In this way, the efficiency, reliability and applicability of data processing can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0033] Figure 1 A flow chart of a data processing method based on the Flink engine provided in this application;
[0034] Figure 2 A schematic diagram of the structure of a data processing device based on the Flink engine provided in this application;
[0035] Figure 3 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] Currently, Flink uses the Kryo serialization framework as a general type serialization mechanism by default. However, in high-concurrency and large-data-volume scenarios, due to the insufficient performance of Kryo, the CPU resources cannot be fully utilized, and Kryo memory allocation and garbage collection may increase the system's memory pressure, especially in long-running stream processing jobs. To this end, this application provides a data processing solution based on the Flink engine, which can effectively improve the efficiency, reliability and applicability of data processing.
[0038] See also Figure 1 As shown, the embodiment of the present invention discloses a data processing method based on the Flink engine, including:
[0039] Step S11: Create a thread-safe instance based on a preset multi-language serialization framework, and register the object to be serialized in the obtained target instance.
[0040] Specifically, in this embodiment, before integration, it is necessary to first create a global thread-safe instance using the preset multi-language serialization framework. That is, the thread-safe instance creation based on the preset multi-language serialization framework and registering the object to be serialized to the obtained target instance includes: creating a global thread-safe Fury (a high-performance multi-language serialization framework) instance to obtain the target instance; registering the object class to be serialized to the target instance. In other words, the preset multi-language serialization framework can be Fury.
[0041] Step S12: Based on a preset integration strategy, the target instance is integrated into the data transmission layer and the state management component in the Flink engine to obtain a corresponding integrated computing engine.
[0042] In this embodiment, after obtaining the target instance, the target instance needs to be integrated into the Flink engine. That is, the preset integration strategy is based on the target instance and the data transmission layer and state management component in the Flink engine are integrated to obtain the corresponding integrated computing engine, including: closing the general serializer in the Flink engine, and integrating the target instance into the data transmission layer, state management component and fault recovery mechanism in the Flink engine to obtain the corresponding integrated computing engine. Among them, closing the general serialization method of the Flink engine will automatically force the Flink engine to adopt the integrated Fury serialization strategy during data processing. In other words, this embodiment integrates the Fury serialization framework into the architecture of the Flink engine to replace the existing Kryo serializer, thereby improving the serialization efficiency. And the integration strategy involves modifications to the data transmission layer, state management component and fault recovery mechanism of the Flink engine to ensure seamless collaboration with the Fury serialization framework.
[0043] Step S13: In the process of data processing using the integrated computing engine, serialization operations and / or deserialization operations are triggered through a preset string compression algorithm, a zero-copy serialization principle, and the target instance to complete the corresponding data transmission operations between engine nodes and the state storage operations of the state backend of the Flink engine.
[0044] In this embodiment, in the process of using the integrated post-computing engine to process data, the object to be serialized in the target instance needs to implement the org apache flink types Value interface in the Flink engine to implement the write and read methods for serialization and deserialization.
[0045] Regarding the optimization of data transmission during data processing, this embodiment will optimize the data transmission process between Flink engine nodes, and use the Fury serialization framework to reduce the serialization and deserialization time. Through Fury's zero-copy technology, the copying of data between the network buffer and the memory is reduced, the network transmission delay is reduced, and the data transmission efficiency is improved. Specifically, the preset string compression algorithm, the zero-copy serialization principle, and the target instance trigger the serialization operation and / or deserialization operation to complete the corresponding data transmission operation between the engine nodes, including: based on the preset string compression algorithm, the preset digital compression algorithm, the zero-copy serialization principle, and the target instance, the data to be transmitted is compressed, and the object to be serialized is automatically called to write the compressed data to be transmitted to complete the corresponding data transmission operation between the engine nodes. That is to say, through the above-mentioned integration method, the Flink engine will automatically adopt the above-mentioned serialization strategy in the entire data processing process, where there is data exchange, so as to optimize the entire data processing process. In the serialization and deserialization process, Fury will automatically use the built-in optimization algorithm and zero-copy technology to reduce the copying of data between the network buffer and the memory, thereby reducing network transmission delay and improving data transmission efficiency. In addition, Fury uses JIT (Just-In-Time Compilation) technology to generate efficient serialization code at runtime, greatly accelerating serialization performance by inlining more methods, optimizing code cache, reducing virtual method calls, conditional branches, hash lookups, metadata writing, and memory reading and writing.
[0046] Furthermore, the compression of the data to be transmitted based on the preset string compression algorithm, the preset digital compression algorithm, the zero-copy serialization principle and the target instance includes: using the meta string encoding strategy (i.e., Meta String Encoding encoding strategy), the zigzag encoding algorithm (i.e., ZigZag algorithm) and the target instance to perform corresponding string compression and / or digital compression on the data to be transmitted to obtain the corresponding compressed data to be transmitted. Among them, the meta string encoding strategy compresses the string, and the encoding is 37.5% more space efficient than UTF-8 (8-bit Universal Character Set / Unicode Transformation Format, a variable-length character encoding method for Unicode). The zigzag encoding algorithm is used to convert signed integers into unsigned integers to reduce the encoding cost of small negative numbers, specifically by converting signed integers into unsigned integers in the manner of (v << 1)^(v >> 31), and then encoding.
[0047] Regarding the serialization of the state backend during data processing, this embodiment improves the state backend of the Flink engine through the above-mentioned integration method, uses the Fury serialization framework to store and restore the computing state, and speeds up the checkpoint. And optimize the serialization process of the state, reduce the time and space required for state storage, and improve the efficiency of state management. Specifically, the Flink engine will store the state of the operator during data processing for fault recovery. These state information also need to be serialized and stored. That is, through the preset string compression algorithm, the zero-copy serialization principle and the target instance triggering the serialization operation and / or deserialization operation to complete the state storage operation of the state backend of the corresponding Flink engine, including: in the process of storing the operator state data of the integrated computing engine to the persistent storage system, based on the preset string compression algorithm, the preset digital compression algorithm, the zero-copy serialization principle and the target instance, the operator state data is compressed, and the object to be serialized is automatically called to write the compressed state data to complete the corresponding serialization operation. That is to say, when persisting these operator states of the Flink engine, it will automatically call the write method (i.e. data writing) implemented by the object class to be serialized in the target instance to serialize the data, thereby using Fury to speed up the entire storage process.
[0048] Further, regarding the fault recovery mechanism in the data processing process, when the fault recovery mechanism of the integrated computing engine is triggered, based on the zero-copy serialization principle and the target instance, the object to be serialized is automatically called to read the compressed state data stored in the persistent storage system to complete the corresponding deserialization operation and obtain the corresponding compressed state data; the compressed state data is decompressed using the preset string compression algorithm and the preset digital compression algorithm to obtain the corresponding operator state data, and the corresponding operator state recovery operation is triggered. In other words, during the recovery process, the operator state will automatically call the read method (i.e., data reading) implemented by the class of the object to be serialized in the target instance to deserialize the data. That is, this embodiment uses the Fury serialization framework to improve the speed and efficiency of Flink fault recovery and shorten the time it takes for the system to recover from a fault. Fast serialization and deserialization capabilities help improve the availability and stability of the system.
[0049] In summary, this embodiment uses the Fury serialization framework to optimize the performance of the Flink engine. Its effectiveness and advantages are reflected in the following aspects:
[0050] 1) Improve serialization efficiency. Since Fury uses JIT dynamic compilation technology to generate optimized code for specific data structures, compared with the Kryo serializer, Fury can reduce the reflection call and type checking overhead in the serialization process, thereby improving the efficiency of data serialization and reducing serialization time.
[0051] 2) Reduce resource consumption. Since Fury uses zero-copy technology, it avoids unnecessary memory allocation and data duplication during serialization and deserialization, thus reducing the frequency of garbage collection (GC) and memory consumption, thereby reducing system resource consumption and improving resource utilization.
[0052] 3) Improve system throughput. Due to the improved serialization efficiency and reduced resource consumption, the Flink engine can process more data more efficiently, improve the overall data processing throughput, and improve the system throughput of the Flink engine.
[0053] 4) Reduce data processing delay. Fury's high efficiency reduces the data transmission time in the network and the processing time in memory, so the delay of data transmission and state management between Flink engine nodes can be reduced. In other words, it can reduce the delay of data processing and improve the response speed of real-time computing.
[0054] 5) Enhance the stability and reliability of the system. Due to the optimization of state management and fault recovery mechanism, the storage and recovery of state are faster and more efficient. Therefore, the system can recover to normal state more quickly when encountering a fault, reducing the impact of the fault on data processing.
[0055] 6) Simplify development and maintenance work. Since Fury can provide a more intuitive and simple API (Application Programming Interface), which is easy to integrate and use, developers can implement data serialization-related functions more quickly and reduce the workload of development and maintenance.
[0056] 7) Improve the flexibility of data processing. Since this embodiment provides an efficient serialization solution that can easily cope with different data processing scenarios and requirements, Flink can more flexibly process various complex data streams and meet diverse business needs. In other words, it improves the flexibility of the Flink engine in processing different data streams.
[0057] Through the above reasoning, it can be concluded that this embodiment has obvious advantages in improving the performance of the Flink engine. These advantages will directly translate into better data processing performance and higher system stability, meeting the strict requirements of real-time big data processing.
[0058] It can be seen that in this application, thread-safe instance creation is performed based on a preset multi-language serialization framework, and the object to be serialized is registered in the obtained target instance; based on a preset integration strategy, the target instance is integrated into the data transmission layer and state management component in the Flink engine to obtain the corresponding integrated computing engine; in the process of using the integrated computing engine for data processing, the serialization operation and / or deserialization operation are triggered by the preset string compression algorithm, the zero-copy serialization principle and the target instance to complete the corresponding data transmission operation between engine nodes and the state storage operation of the state backend of the Flink engine. That is, in this embodiment, by creating a thread-safe instance, the preset multi-language serialization framework is integrated into the data transmission layer and state management component in the Flink engine, and then the obtained integrated computing engine is used for data processing, and when performing data transmission operations between engine nodes and state storage operations of the engine state backend, the created target instance is used to trigger serialization or deserialization to complete data processing. In this way, the efficiency, reliability and applicability of data processing can be effectively improved.
[0059] See also Figure 2 As shown, the embodiment of the present application also discloses a data processing device based on the Flink engine, including:
[0060] An instance creation module 11 is used to create a thread-safe instance based on a preset multi-language serialization framework, and register the object to be serialized into the obtained target instance;
[0061] An instance integration module 12 is used to integrate the target instance into the data transmission layer and state management component in the Flink engine based on a preset integration strategy to obtain a corresponding integrated computing engine;
[0062] The data processing module 13 is used to trigger serialization operations and / or deserialization operations through a preset string compression algorithm, a zero-copy serialization principle, and the target instance during data processing using the integrated post-computing engine, so as to complete the corresponding data transmission operations between engine nodes and the state storage operations of the state backend of the Flink engine.
[0063] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0064] It can be seen that in this application, thread-safe instance creation is performed based on a preset multi-language serialization framework, and the object to be serialized is registered in the obtained target instance; based on a preset integration strategy, the target instance is integrated into the data transmission layer and state management component in the Flink engine to obtain the corresponding integrated computing engine; in the process of using the integrated computing engine for data processing, the serialization operation and / or deserialization operation are triggered by the preset string compression algorithm, the zero-copy serialization principle and the target instance to complete the corresponding data transmission operation between engine nodes and the state storage operation of the state backend of the Flink engine. That is, in this embodiment, by creating a thread-safe instance, the preset multi-language serialization framework is integrated into the data transmission layer and state management component in the Flink engine, and then the obtained integrated computing engine is used for data processing, and when performing data transmission operations between engine nodes and state storage operations of the engine state backend, the created target instance is used to trigger serialization or deserialization to complete data processing. In this way, the efficiency, reliability and applicability of data processing can be effectively improved.
[0065] In some specific embodiments, the instance creation module 11 may specifically include:
[0066] The target instance acquisition unit is used to create a global thread-safe Fury instance to obtain the target instance;
[0067] The object registration unit is used to register the object to be serialized in the target instance.
[0068] In some specific embodiments, the instance integration module 12 may specifically include:
[0069] The instance integration unit is used to close the universal serializer in the Flink engine and integrate the target instance into the data transmission layer, state management component and fault recovery mechanism in the Flink engine to obtain the corresponding integrated computing engine.
[0070] In some specific embodiments, the data processing module 13 may specifically include:
[0071] The data transmission unit is used to compress the data to be transmitted based on a preset string compression algorithm, a preset digital compression algorithm, a zero-copy serialization principle and the target instance, and automatically call the object to be serialized to write the compressed data to be transmitted to complete the data transmission operation between the corresponding engine nodes.
[0072] In some specific embodiments, the data transmission unit may specifically include:
[0073] The data compression subunit is used to perform corresponding string compression and / or digital compression on the data to be transmitted by using the meta-string encoding strategy, the zigzag encoding algorithm and the target instance to obtain the corresponding compressed data to be transmitted.
[0074] In some specific embodiments, the data processing module 13 may specifically include:
[0075] A state data storage unit is used to compress the operator state data of the integrated computing engine based on a preset string compression algorithm, a preset digital compression algorithm, a zero-copy serialization principle and the target instance during the process of storing the operator state data of the integrated computing engine into a persistent storage system, and automatically call the object to be serialized to write data to the obtained compressed state data to complete the corresponding serialization operation.
[0076] In some specific embodiments, the data processing device based on the Flink engine may further include:
[0077] A state data acquisition unit, configured to automatically call the object to be serialized to read the compressed state data stored in the persistent storage system based on the zero-copy serialization principle and the target instance when the fault recovery mechanism of the integrated computing engine is triggered, so as to complete the corresponding deserialization operation and obtain the corresponding compressed state data;
[0078] The state recovery unit is used to decompress the compressed state data by using the preset string compression algorithm and the preset digital compression algorithm to obtain the corresponding operator state data and trigger the corresponding operator state recovery operation.
[0079] Furthermore, the present application also discloses an electronic device. Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0080] Figure 3 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data processing method based on the Flink engine disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0081] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0082] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0083] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data processing method based on the Flink engine executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.
[0084] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the data processing method based on the Flink engine disclosed above is implemented. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.
[0085] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0086] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0087] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0088] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0089] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data processing method based on the Flink engine, characterized in that: include: Create thread-safe instances based on the preset multi-language serialization framework, and register the object to be serialized to the obtained target instance; Based on a preset integration strategy, the target instance is integrated into the data transmission layer and the state management component in the Flink engine to obtain a corresponding integrated computing engine; In the process of using the integrated post-computing engine to process data, serialization operations and / or deserialization operations are triggered through a preset string compression algorithm, a zero-copy serialization principle, and the target instance to complete the corresponding data transmission operations between engine nodes and the state storage operations of the state backend of the Flink engine.
2. The data processing method based on the Flink engine according to claim 1, characterized in that: The thread-safe instance creation based on the preset multi-language serialization framework and registering the object to be serialized to the obtained target instance includes: Create a global thread-safe Fury instance to get the target instance; Register the object to be serialized with the target instance.
3. The data processing method based on the Flink engine according to claim 1, characterized in that: The method of integrating the target instance into the data transmission layer and the state management component in the Flink engine based on the preset integration strategy to obtain the corresponding integrated computing engine includes: The general serializer in the Flink engine is closed, and the target instance is integrated into the data transmission layer, state management component, and fault recovery mechanism in the Flink engine to obtain a corresponding integrated computing engine.
4. The data processing method based on the Flink engine according to any one of claims 1 to 3, characterized in that: The serialization operation and / or deserialization operation is triggered by the preset string compression algorithm, the zero-copy serialization principle and the target instance to complete the data transmission operation between the corresponding engine nodes, including: Based on a preset string compression algorithm, a preset digital compression algorithm, a zero-copy serialization principle and the target instance, the data to be transmitted is compressed, and the object to be serialized is automatically called to write the compressed data to be transmitted, so as to complete the data transmission operation between the corresponding engine nodes.
5. The data processing method based on the Flink engine according to claim 4 is characterized in that: The compressing the data to be transmitted based on the preset string compression algorithm, the preset digital compression algorithm, the zero-copy serialization principle and the target instance includes: The meta-string encoding strategy, the zigzag encoding algorithm and the target instance are used to perform corresponding string compression and / or digital compression on the data to be transmitted, so as to obtain the corresponding compressed data to be transmitted.
6. The data processing method based on the Flink engine according to claim 1, characterized in that: The serialization operation and / or deserialization operation are triggered by the preset string compression algorithm, the zero-copy serialization principle, and the target instance to complete the state storage operation of the state backend of the corresponding Flink engine, including: In the process of storing the operator state data of the integrated computing engine to the persistent storage system, the operator state data is compressed based on a preset string compression algorithm, a preset digital compression algorithm, a zero-copy serialization principle and the target instance, and the object to be serialized is automatically called to write data to the obtained compressed state data to complete the corresponding serialization operation.
7. The data processing method based on the Flink engine according to claim 6, characterized in that: Also includes: When the fault recovery mechanism of the integrated computing engine is triggered, the object to be serialized is automatically called based on the zero-copy serialization principle and the target instance to read the compressed state data stored in the persistent storage system to complete the corresponding deserialization operation and obtain the corresponding compressed state data; The compressed state data is decompressed using the preset string compression algorithm and the preset digital compression algorithm to obtain the corresponding operator state data, and trigger a corresponding operator state recovery operation.
8. A data processing device based on the Flink engine, characterized in that: include: An instance creation module is used to create thread-safe instances based on a preset multi-language serialization framework and register the object to be serialized to the obtained target instance; An instance integration module is used to integrate the target instance into the data transmission layer and state management component in the Flink engine based on a preset integration strategy to obtain a corresponding integrated computing engine; The data processing module is used to trigger serialization and / or deserialization operations through a preset string compression algorithm, a zero-copy serialization principle, and the target instance during data processing using the integrated post-computing engine, so as to complete the corresponding data transmission operations between engine nodes and the state storage operations of the state backend of the Flink engine.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data processing method based on the Flink engine according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the data processing method based on the Flink engine according to any one of claims 1 to 7.