A data stream batch processing method, device, equipment and medium

Abstract

The application relates to the technical field of data processing, in particular to a data stream batch processing method and device, equipment and medium. The method comprises the following steps: mapping an input data source table into a corresponding object relationship mapping class based on an object relationship mapping rule; after attribute assignment operation is performed on the object relationship mapping class, the operation method of data stream batch processing is woven to obtain a rule proxy class with an operation method in an object-oriented programming language; the rule proxy class is bridged to satisfy a calling class of a conversion service interface to obtain a bridged calling class, so that a preset conversion service interface calls the bridged calling class to perform conversion processing on data in the data source table. In this way, a SQL script statement does not need to be defined, relevant entity classes and attributes can be directly defined according to business requirements, the structure of the data source table is described through the object relationship mapping class, and then the data source table is abstracted, so that conversion processing can be performed on data in the data source table.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data stream batch processing method and device, equipment and medium. BACKGROUND

[0002] Flink is a distributed and high-performance stream processing application framework, which not only supports high throughput and exactly-once semantics real-time computing, but also supports batch processing based on stream computing combined with data-defined windows. Flink uses a state mechanism to implement a stateful stream computing engine. Flink stream computing uses state storage nodes to store intermediate results, and can also implement fault recovery through a checkpoint mechanism.

[0003] The basis of the Flink real-time computing model is stream and transform, and the basic flow implementation is that each data stream starts from one or more data sources, is processed through a number of transforms, and ends at the output of the computing result (sink).

[0004] Flink SQL (Structured Query Language) is a simplified Flink real-time computing model that uses a development language that complies with standard SQL semantics to implement obtaining raw data sources, applying computing logic (transform) for computing and summarizing, and outputting results (sink) to a specified target.

[0005] However, Flink SQL is written using SQL semantics, and SQL writing requires certain database query and operation experience, so non-database developers or business program developers cannot directly define the required SQL script statements according to business requirements, which is not conducive to business implementation. SUMMARY

[0006] The present application provides a data stream batch processing method, device, equipment and medium, which abstracts a data source table according to business requirements without defining SQL script statements, so as to implement transform processing of data in the data source table.

[0007] The specific technical solutions provided by the embodiments of the present application are as follows:

[0008] In a first aspect, the embodiments of the present application provide a data stream batch processing method, comprising:

[0009] mapping the input data source table into a corresponding object relationship mapping class based on an object relationship mapping rule, wherein the object relationship mapping class refers to a class defined by an object-oriented programming language;

[0010] After attribute assignment operation is performed on the object-relational mapping class, an operation method of data stream batch processing is woven to obtain a rule agent class with the operation method in the object-oriented programming language;

[0011] The rule agent class is bridged to an invocation class satisfying a conversion service interface to obtain a bridged invocation class, so that the preset conversion service interface invokes the bridged invocation class to perform conversion processing on data in the data source table.

[0012] In an optional implementation, the object-relational mapping rule is used to map the input data source table into a corresponding object-relational mapping class, including:

[0013] Based on the object-relational mapping rule, a class name corresponding to a table name of the input data source table and a class attribute name corresponding to a table field name of the data source are determined.

[0014] The table name and the corresponding class name and the table field name and the corresponding class attribute name are mapped and associated in an annotation manner.

[0015] In an optional implementation, after attribute assignment operation is performed on the object-relational mapping class, an operation method of data stream batch processing is woven to obtain a rule agent class with the operation method in the object-oriented programming language, including:

[0016] The object-relational mapping class is instantiated using a reflection mechanism of the object-oriented programming language, and attribute assignment operation is performed on the instantiated object-relational mapping class.

[0017] The operation method of data stream batch processing is woven into the object-relational mapping class after attribute assignment through a dynamic proxy manner to obtain a rule agent class with the operation method in the object-oriented programming language.

[0018] In an optional implementation, the rule agent class is bridged to an invocation class satisfying a preset conversion service to obtain a bridged invocation class, including:

[0019] Based on the operation method of the rule agent class, related constructor methods, assembly methods, and destructor methods satisfying conversion calls of the preset conversion service are added in a bridging mode.

[0020] Based on the rule agent class after the related constructor methods, assembly methods, and destructor methods are added, the bridged invocation class is obtained.

[0021] In an optional implementation, the method further includes:

[0022] If the input data source table does not satisfy the preset condition, the data source table is added with annotation information in an annotation manner, and the annotation information is used to indicate that the data source table is expressed by using a SQL query statement.

[0023] The data source table added with the annotation information is input into a SQL parsing engine for conversion processing.

[0024] In a second aspect, an embodiment of the present application provides a data stream batch processing apparatus, comprising:

[0025] A mapping module is configured to map an input data source table into a corresponding object-relational mapping class based on an object-relational mapping rule, wherein the object-relational mapping class refers to a class defined by an object-oriented programming language.

[0026] A rule proxy module is configured to weave an operation method of data stream batch processing into the object-relational mapping class after attribute assignment operation, so as to obtain a rule proxy class with the operation method in the object-oriented programming language.

[0027] A bridge construction module is configured to bridge the rule proxy class to satisfy a calling class of a conversion service interface, so as to obtain a bridge calling class, and enable the preset conversion service interface to call the bridge calling class to perform conversion processing on data in the data source table.

[0028] In an optional embodiment, the mapping module is specifically configured to:

[0029] Determine a class name corresponding to a table name of the input data source table and a class attribute name corresponding to a table field name of the data source based on an object-relational mapping rule.

[0030] Map and associate the table name and the corresponding class name and the table field name and the corresponding class attribute name in an annotation manner.

[0031] In an optional embodiment, the rule proxy module is specifically configured to:

[0032] Use a reflection mechanism of an object-oriented programming language to perform instantiation operation on the object-relational mapping class and perform attribute assignment operation on the instantiated object-relational mapping class.

[0033] Weave the operation method of data stream batch processing into the object-relational mapping class after attribute assignment operation in a dynamic proxy manner, so as to obtain the rule proxy class with the operation method in the object-oriented programming language.

[0034] In an optional embodiment, the bridge construction module is specifically configured to:

[0035] The bridge mode is adopted, and on the basis of the operation method of the rule proxy class, the related construction method, the assembly method and the destruction method satisfying the preset conversion service conversion call are added;

[0036] Based on the rule proxy class after adding the related construction method, the assembly method and the destruction method, the bridge call class is obtained.

[0037] In an optional implementation, the apparatus further includes an annotation module configured to:

[0038] If the input data source table does not satisfy the preset condition, annotation information is added to the data source table by an annotation method, and the annotation information is used to indicate that the SQL query statement is used for expression.

[0039] The data source table after adding the annotation information is input into a SQL parsing engine for conversion processing.

[0040] In a third aspect, an electronic device is provided, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any method in the first aspect.

[0041] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of any method in the first aspect are implemented.

[0042] In a fifth aspect, a computer program product is provided, which includes a computer program stored in a computer readable storage medium, and when a processor of an electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any method in the first aspect.

[0043] The embodiments of the present application have at least the following beneficial effects:

[0044] In the scheme of the embodiment of the present application, the input data source table is mapped into a corresponding object-relational mapping class based on an object-relational mapping rule; after attribute assignment operation is performed on the object-relational mapping class, the operation method of data stream batch processing is woven, to obtain a rule proxy class with an operation method in an object-oriented programming language; the rule proxy class is bridged to satisfy the calling class of the conversion service interface, to obtain a bridged calling class, so that the preset conversion service interface calls the bridged calling class to perform conversion processing on the data in the data source table. In this way, by using the object-oriented idea of object-relational mapping, the data source table is mapped into an object form through metadata (information describing data attributes), object-oriented programming language object operation is realized by converting Flink SQL operation into object operation, that is, on the basis of the existing layered Flink stream batch data operation SQL level, an object-relational mapping layer operation is added through the object-relational mapping rule, so that the SQL script statement does not need to be defined, and the related entity class and attribute can be directly defined according to the business requirement, the structure of the data source table is described through the object-relational mapping class, and then the abstraction of the data source table is realized, so as to realize the conversion processing of the data in the data source table.

[0045] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by means of the structures particularly pointed out in the written description and claims hereof as well as in the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0047] Figure 1 A flowchart of a data stream batch processing method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the level of an object-relational mapping operation provided in an embodiment of the present application;

[0049] Figure 3 A logic schematic diagram of a data stream batch processing method provided in an embodiment of the present application;

[0050] Figure 4 A specific implementation process schematic diagram of a data stream batch processing method provided in an embodiment of the present application;

[0051] Figure 5This is a structural block diagram of a data stream batch processing device provided in the embodiments of this application;

[0052] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0054] Stream processing, or data stream processing for short, refers to the process where, after a piece of data is processed, it is serialized and stored in a buffer, and then immediately transmitted over the network to the next node for further processing. In terms of scope, data stream processing is an unbounded, continuous data processing procedure.

[0055] Batch processing, or simply data batch processing, refers to a process where, after a piece of data is processed, it is serialized and stored in a cache, but not immediately transmitted over the network to the next node. When the cache is full, the data is persistently stored on disk. Only after all data has been processed does the data begin to be transmitted over the network to the next node. From a scope perspective, batch processing is a bounded data processing procedure. It can be understood as a special case of stream processing. In stream processing, a sliding or scrolling window is defined, and results are generated each time the window slides or scrolls. Batch processing, on the other hand, defines a global window, to which all records belong. Stream processing can be understood as batch processing implemented through defined windows.

[0056] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more.

[0057] The design concept of this application is briefly outlined below.

[0058] Flink SQL is a simplified Flink real-time computing model, using a development language that complies with standard SQL semantics to achieve obtaining a raw data source (source), applying computing logic (transform) for computing and summarizing, and outputting (Sink) results to a specified target. According to the above description, a complete Flink SQL includes the following three parts:

[0059] 1. Data source processor (Source Operator): Abstracts external data sources and unifies data source acquisition methods;

[0060] 2. Transformation processor (Transformation Operator): Also known as operator operation. Realizes query, aggregation operation, and operations such as union, join, projection, difference, intersection, and window for SQL;

[0061] 3. Output processor (Sink Operator): Outputs the operation result to a specified target service. Integrates the abstraction of a plurality of result table output forms.

[0062] However, Flink SQL uses SQL semantics for writing, and has the following disadvantages in application:

[0063] 1. The abstraction granularity of the data source only reaches the data collection level;

[0064] 2. SQL writing requires certain database query and operation experience, and non-database developers or business program developers cannot directly define the required SQL script statements according to business requirements;

[0065] 3. Non-object-oriented. SQL is a data query-oriented description syntax, and is not associated with object-oriented programming ideas;

[0066] 4. Easy to produce hard coding, high complexity, and not conducive to later maintenance.

[0067] 5. Parameter assignment is based on simple encapsulation, and cannot be combined with business models.

[0068] In view of this, this application provides a data stream batch processing method, apparatus, device, and medium. Using object-oriented principles through object-relational mapping (ORM), it maps data source tables to object forms through metadata (information describing data attributes). This transforms Flink SQL operations into object operations in an object-oriented programming language. Specifically, it adds an ORM layer to the existing layered SQL layer of Flink stream batch data operations. Thus, when implementing real-time computing streams or batch processing business requirements, there is no need to define SQL script statements. Relevant entity classes and attributes can be directly defined according to business needs. The ORM class describes the structure of the data source table, thereby abstracting the data source table to facilitate data transformation and processing. Furthermore, mapping the data source table to object forms provides a smaller granularity of abstraction, facilitating later maintenance and integration with business models.

[0069] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0070] The data stream batch processing method of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] The data stream batch processing in this application can be executed by a server containing Flink SQL. This server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0072] like Figure 1 As shown, a data stream batch processing method provided in an embodiment of this application includes the following steps S101-S103:

[0073] Step S101: Based on the object-relational mapping rules, the input data source table is mapped to the corresponding object-relational mapping class. The object-relational mapping class refers to the class defined by the object-oriented programming language.

[0074] The object relational mapping (ORM) is a program design technique for realizing conversion between data of different types of systems in an object-oriented programming language. In effect, it actually creates a "virtual object database" that can be used in a programming language. In the embodiments of the present application, an object class corresponding to each attribute of a data source table can be created in advance to form an object relational mapping rule.

[0075] The data source table can include a stream table and a dimension table. The dimension table refers to a dimension information table, wherein the dimension attribute is an angle of observing data, and supplements information of the fact table. The stream table refers to a table mapped from real-time stream data. In a join query, each incoming data is actively queried in the dimension table to determine whether there is matching data.

[0076] In an optional embodiment, the step S101 can include the following steps A1-A2.

[0077] A1, determining, based on the object relational mapping rule, a class name corresponding to a table name of the input data source table and a class attribute name corresponding to a table field name of the data source.

[0078] A2, mapping and associating the table name and the corresponding class name and the table field name and the corresponding class attribute name by an annotation method.

[0079] The object relational mapping rule includes a mapping relationship between the class name and the table name and between the class attribute and the table field name. Based on the object relational mapping rule, the class name corresponding to the table name of the data source table and the class attribute name corresponding to the table field name of the data source can be determined.

[0080] For example, in the dimension table shown in Table 1, the table name is "storage", and the table field names include the table identifier field name "id" and the table column field name "item_name". For another example, in the stream table shown in Table 2, the table name is "sample", and the table field names include the table identifier field name "id" and the table column field name "item_name".

[0081] Table 1 storage

[0082] id item_name 1 CPU 2 memory

[0083] Table 2

[0084] sample {"id”:"1”,"item_name”:CPU} {"id”:"2”,"item_name”:memory}

[0085] Specifically, Java annotation can be used to describe the association information between each attribute of the data source table and the corresponding object class, i.e., object relationship mapping definition information, which can be parsed and processed by the relationship mapping component through reflection, and the result is a class with object relationship mapping definition information. For example, Table 3 shows the object relationship mapping rule.

[0086] Table 3

[0087]

[0088] For example, the dimension table in Table 1 is mapped and associated with the object relationship mapping rule in Table 3 to obtain the following object relationship mapping definition information:

[0089] @Entity(name=“storage”)

[0090] public class Storage{

[0091] @Id

[0092] private Long id;

[0093] @Column(“item_name”)

[0094] Private String itemName;

[0095] public Storage(){} ...

[0097] }

[0098] For another example, the flow table in Table 2 is mapped and associated with the object relationship mapping rule in Table 3 to obtain the following object relationship mapping definition information:

[0099] @Entity(name=“sample”)

[0100] public class Storage{

[0101] @Id

[0102] private Long id;

[0103] @Column(“item_name”)

[0104] Private String itemName;

[0105] public Storage(){} ...

[0107] }

[0108] Step S102, after the attribute assignment operation of the object relationship mapping class, the operation method of the data stream batch processing is woven to obtain the rule agent class with the operation method of the object-oriented programming language.

[0109] The attributes in the object relationship mapping class are assigned, for example, the field name "item_name" in the above table 1 corresponds to the attribute "Column" in the object relationship mapping class. As can be seen from the above table 1, the value corresponding to the attribute "Column" includes "CPU" and "memory".

[0110] The operations of the data stream batch processing include but are not limited to retrieval, update, batch addition, etc. The above operation methods can be woven into the object relationship mapping class by dynamic proxy method to form a rule agent class that can interact with the Flink DataTable (batch data form of DataStream).

[0111] Dynamic proxy refers to a technology of creating a proxy object of a target object during program running and performing functional enhancement on the methods in the target object. In the process of generating the proxy object, the target object does not change, and the methods in the proxy object are enhanced methods of the methods of the target object. It can be understood as dynamic interception of the methods in the target object during running, and performing functional operations before and after the intercepted methods. The target object can be understood as the object relationship mapping class in the embodiments of the present application.

[0112] In an optional embodiment, step S102 can specifically include the following steps B1-B2:

[0113] B1, using the reflection mechanism of the object-oriented programming language, the object relationship mapping class is instantiated, and the instantiated object relationship mapping class is subjected to attribute assignment operation.

[0114] Specifically, using the reflection mechanism of the object-oriented programming language, through dynamic analysis, referring to the object relationship mapping definition, the object relationship mapping class can be instantiated and the attribute assignment operation can be completed.

[0115] Reflection can dynamically create an instance of any class, call the methods of the class or instance, or change the properties of the class or instance through the reflection method. Reflection can also access the internal information of the type, including the modifier, field, and method of the type. Through reflection, the class can be assembled at runtime without source code linking between components, which reduces the code coupling degree and can also be used as the basis for dynamic proxy operation.

[0116] B2, by means of dynamic proxy, weaving the operation method of data stream batch processing in the object relationship mapping class after attribute assignment, obtaining the rule proxy class of object-oriented programming language with operation method.

[0117] Among them, the operation of data stream batch processing includes retrieval, update, batch addition, etc. Operation, corresponding to Select(), Update() and BatchInsert() operation method in turn, the above operation method is woven into the object relationship mapping class by dynamic proxy, forming a rule proxy class that can interact with Flink DataTable.

[0118] The rule proxy has a unified call entry method: invoke(Object proxy,Method,Object[]args), which can let the Flink Table API conversion service as the caller pass in the specified parameters to realize the consistent Flink Table API call.

[0119] Step S103, bridge the rule proxy class to meet the call class of the conversion service interface, obtain the bridged call class, so that the preset conversion service interface calls the bridged call class to convert the data in the data source table.

[0120] Among them, through the bridging mode, the rule proxy class can be converted into a bridged call class that can be converted by the preset conversion service (FlinkTable) as an input source. The bridged call class supports conversion from the preset conversion service interface (FlinkTable API) to SQL, which can complete the subsequent data stream batch processing operation.

[0121] In an optional implementation, the above step S103 can specifically include the following steps C1-C2:

[0122] C1, using a bridging mode, adding related construction methods, assembly methods and destruction methods that meet the preset conversion service conversion call on the basis of the operation method of the rule proxy class.

[0123] Among them, the bridging mode separates the abstract part from its implementation part, so that they can be changed independently. It is an object structural pattern, also known as Handle and Body pattern or interface pattern. Using the bridging mode, the rule proxy class is bridged to meet the call class of the Flink Table API conversion service, that is, on the basis of the original Flink DataTable proxy operation method, the related construction, assembly and destruction methods that meet the Flink Table API conversion call are added.

[0124] The bridging mode can separate the Flink Table API operation from the Flink DataTable rule agent, that is, can keep the two independent, or can contact the two through the bridging method. The bridging mode decouples the inherent binding relationship between the abstraction and the implementation using the "association relationship between objects", so that the abstraction and the implementation can change along their respective dimensions. The change of the abstraction and the implementation along their respective dimensions means that the abstraction and the implementation are no longer in the same inheritance hierarchy, but "subclass" them so that each of them has its own subclass, so as to obtain a multi-dimensional combined object by combining any subclass. In many cases, the bridging mode can replace the multi-layer inheritance scheme, which violates the "single responsibility principle", has poor reusability, and has a large number of classes. The bridging mode is a better solution than the multi-layer inheritance scheme, which greatly reduces the number of subclasses. The bridging mode improves the scalability of the system, and any expansion of a dimension in the two change dimensions does not need to modify the original system, which meets the "open-closed principle".

[0125] C2, based on the rule agent class after adding the related construction method, assembly method and destruction method, obtain the bridging call class.

[0126] After the rule agent class is assembled through the bridging mode, the bridging call class is obtained, that is, the call class that meets the Flink Table API, so that the Flink Table API can call the bridging call class to perform data conversion and processing on the data in the data source table.

[0127] In the embodiment of the application, the object-oriented idea is used through object relationship mapping, the data source table is mapped to an object form through metadata, the Flink SQL operation is converted into an object operation of an object-oriented programming language, that is, on the basis of the existing layered SQL level of the Flink stream batch data operation, an object relationship mapping layer operation is added through an object relationship mapping rule. In this way, when realizing the business requirements of real-time computing stream or batch processing, it is not necessary to define the SQL script statement, and the related entity class and attribute can be directly defined according to the business requirements, the structure of the data source table is described through the object relationship mapping class, and then the abstraction of the data source table is realized, so as to realize the conversion processing of the data in the data source table. Moreover, the data source table is mapped to an object form, the abstraction granularity is small, the later maintenance is facilitated, and the business model can be combined.

[0128] In an optional implementation, if the input data source table does not meet the preset condition, annotation information is added to the data source table through an annotation mode, the annotation information is used to indicate that the SQL query statement is used for expression; and the data source table after adding the annotation information is input into a SQL parsing engine for conversion processing.

[0129] Specifically, the preset condition could be that the data source table has a complex structure and cannot be mapped to the corresponding object-relational mapping class based on object-relational mapping rules. In this case, annotations can be used to instruct the data source table to be expressed using SQL query statements, and the SQL parsing engine (Flink SQL parsing engine) can be used directly for conversion.

[0130] For example, the following is an example of using the @Query annotation to instruct the data source table to be represented by an SQL query statement:

[0131] @Entity(name="processe")

[0132] public class Processe{

[0133] @Query("select name from process")

[0134] Private String querystr;

[0135] public Storage(){} ...

[0137] }

[0138] In the above embodiments of this application, for semantic descriptions of complex relationships, the @Query annotation is provided to directly input Flink SQL, which is then directly submitted to the Flink Table SQL parsing engine for processing.

[0139] The following is combined with Figure 2 - Figure 4 The specific implementation process of the data stream batch processing method in the embodiments of this application is described.

[0140] like Figure 2 As shown, this application uses object-oriented principles through Object-Relational Mapping (ORM) to map real-time stream tables (DataStream Tables) and dimension tables (DataSet Tables) into objects via metadata, thus transforming Flink SQL operations into object operations in an object-oriented programming language. In other words, it adds an object-relational mapping layer to the existing layered SQL layer of Flink stream and batch data operations through object-relational mapping rules.

[0141] The overall flow of the data stream batch processing method in this application embodiment is as follows: Figure 3As shown, for the input stream batch data DataStream or DataSet, the input stream batch data object relationship set is obtained through object relationship mapping, the input DataTable is obtained through rule agent, and then the bridge structure is constructed to input Flink Table Transform for conversion processing, output DataTable, and finally output stream batch data DataStream or DataSet.

[0142] The overall flow in Figure 3 will be described in detail below.

[0143] As shown in Figure 4 , the embodiment of the present application defines a document through object relationship mapping to realize object relationship mapping conversion of stream batch data source; through a rule agent, a processing method combined with the object relationship mapping relationship is assembled to map the input stream batch data into a rule agent type with an operation method in an object-oriented programming language; through a bridge structure, the object instance set is converted into a DataTable parameter object instance as an input source that can be converted by Flink Table, and the object instance supports conversion from Flink Table API to SQL, so that subsequent data stream batch processing operations can be completed.

[0144] The data stream batch processing device based on the object relationship mapping rule generally includes the following document definitions:

[0145] Data source configuration information document: connection information required when connecting with a specified data source. Including: database, database table name, login name, password and connection string.

[0146] Data source configuration document example:

[0147] stream.src.name=kafka_stream_source

[0148] stream.src.ip=10.0.0.1

[0149] stream.src.port=3333

[0150] stream.src.topic=stream_source

[0151] stream.dst.name=kafka_dest_store

[0152] stream.dst.ip=10.1.2.2

[0153] stream.dst.port=4444

[0154] stream.dst.topic = dest_store

[0155] db.name = mysql_db

[0156] db.url = jdbc:mysql: / / 10.0.0.1 / test_db

[0157] db.username = root

[0158] db.password = root123

[0159] db.table_name = configuration

[0160] Specifically, stream represents a "stream table", and db represents a "dimension table"; the stream table corresponds to a source "src" and a target "dst"; the dimension table only supports association queries, i.e., as a supplement to the association of stream table data.

[0161] 1. Data stream table or dimension table and object relationship mapping definition: refers to the mapping relationship definition configuration between the data stream table or dimension table and the class of the object-oriented programming language, i.e., the relationship configuration of identifying the class name corresponding to the table name and the class attribute corresponding to the table field name. The implementation method is:

[0162] a) Java annotation is used to realize the mapping definition information of the object association class and the source table;

[0163] b) The object association mapping definition information is parsed and processed by the relational mapping component through reflection;

[0164] c) The output result is a class with object association mapping configuration relationship definition.

[0165] 2. Class defined by object-oriented programming language: used to describe the structure of the data source table (stream table or dimension table), with one or more identified attribute fields and several data attribute fields, corresponding to the record mapping in the data stream table or dimension table.

[0166] The naming specification of the object-oriented class definition is as follows:

[0167] a) One class corresponds to one table. For example, the class name can be in English singular, and the first letter is capitalized; the table name corresponds to the specific naming, which can also be defined as an alias, and the format should be in lowercase.

[0168] b) Each table must have a primary key field, usually named id, which is an integer field.

[0169] c) The foreign key field name rule of the table association is: the associated primary table name_id, such as: item_id. The implementation process of the data stream batch processing method based on the object relationship mapping rule is as follows:

[0170] a) The object class is mapped and associated with the data stream table and the field of the dimension table through the annotation method:

[0171] b) According to the known table name and the field name and type contained in the table, the corresponding annotations are defined, for example, Table 3 in the above embodiment of the application;

[0172] c) The data stream table or the dimension table data definition class is applied according to the annotation;

[0173] d) Directly annotated with a SQL query statement.

[0174] 2. Using the reflection mechanism of the object-oriented programming language, the stream table or the dimension table data class is instantiated through dynamic analysis, and the attribute assignment operation is completed by referring to the object relationship mapping definition.

[0175] Reflection can dynamically create instances, call methods of classes or instances, or change properties of classes or instances through reflection methods in a running state. Reflection can also access internal information of types, including modifiers, fields, methods, etc. of types. Through reflection, classes can be assembled at runtime without source code linking between components, reducing code coupling, and can also be used as the basis for dynamic proxy operation implementation.

[0176] 3. Using a dynamic proxy method to weave the object relationship mapping class related data stream batch processing related predicate operation method logic.

[0177] The search, update, and batch insertion operations required by the object relationship mapping of the stream table or the dimension table are respectively woven into the Select(), Update(), and BatchInsert() operation methods of the object relationship mapping class through a dynamic proxy method, forming a rule proxy class that can interact with the Flink DataTable.

[0178] The rule proxy implementation class has a unified call entry method: invoke(Object proxy, Method, Object[] args)

[0179] The method can make the Flink Table API conversion service as a caller to pass in specified parameters to achieve consistent Flink Table API calls.

[0180] 4. Bridge the rule agent operation implementation class to meet the calling class of the Flink Table API conversion service using the bridge mode, that is, add related construction, assembly and destruction methods that meet the Flink Table API conversion call on the basis of the original Flink DataTable agent operation method.

[0181] The bridge mode can separate the Flink Table API operation from the Flink DataTable rule agent, that is, the two can remain independent of each other, or the two can be connected through the bridge method.

[0182] 5. The object relationship mapping rule agent class assembled through the bridge mode, and the output result of the data stream batch processing through the Flink Table API conversion processing can be:

[0183] (1) meet the Flink Table API calling class

[0184] (2) Flink SQL as an intermediate state

[0185] (3) error information of incorrect parsing.

[0186] The object relationship mapping data stream batch processing device preferentially calls the bridge service class for the Flink Table API conversion processing according to the object relationship mapping processing; for complex association relationship semantic description, the @Query annotation is provided to directly input the Flink SQL, and the SQL is directly submitted to the Flink Table SQL parsing engine for processing.

[0187] The embodiment of the application utilizes the object-oriented language to be closer to the business angle, realizes the data stream batch processing operation through the object relationship rule; avoids writing the SQL statement, and when realizing the real-time computing stream or batch processing business demand, the related entity class and attribute can be directly defined according to the business demand, the association relationship between the data stream table and the dimension table is described through the combination relationship of multiple class instances; the object relationship mapping is configured in the annotation mode, and the number of related configuration files is reduced. The selection is flexible, and the object relationship mapping rule agent data stream batch operation can be supported, and the complex query operation mode of directly inputting the Flink SQL can also be supported.

[0188] Compared with the prior art, the data stream batch processing method of the embodiment of the application has the following advantages:

[0189] 1. Compared with the basic data set based data stream batch operation, the embodiment of the application realizes a data stream batch processing mechanism based on the object relationship mapping rule;

[0190] 2. Compared with the Flink Table API data stream batch processing, only the query operation can be performed through the SQL mode, and the embodiment of the present application provides an object-relational mapping rule agent class, a bridging calling class conforming to the Flink Table API conversion is assembled, and the data stream batch processing operation is realized;

[0191] 3. Compared with the difference in semantics and description between the SQL writing and the object-oriented programming language, the embodiment of the present application converts the Flink stream batch data from the input end into the object-oriented relational mapping structure through the object-relational mapping processing, supports the operation mode of the programming language, that is, the operation stream batch data is assembled through the object instance, attribute assignment and method calling mode;

[0192] 4. Compared with the object-relational mapping that cannot meet the object-relational description definition of the complex scene, the embodiment of the present application supports the data stream batch operation through the object-relational mapping mode, and also supports the data stream batch operation through the input Flink SQL directly transmitted to the Flink Table API.

[0193] Based on the same inventive concept, the embodiment of the present application also provides a data stream batch processing device. Since the principle of the device for solving the problem is similar to the above-mentioned data stream batch processing method, the implementation of the device can be referred to the embodiments of the method, and the repeated parts will not be described here.

[0194] As shown in Figure 5 , the embodiment of the present application provides a data stream batch processing device, which comprises:

[0195] The mapping module 51 is configured to map the input data source table into a corresponding object-relational mapping class based on an object-relational mapping rule, and the object-relational mapping class is a class defined by an object-oriented programming language;

[0196] The rule agent module 52 is configured to weave the operation method of the data stream batch processing into the object-relational mapping class after the attribute assignment operation, and obtain a rule agent class with the operation method of the object-oriented programming language;

[0197] The bridging construction module 53 is configured to bridge the rule agent class to satisfy the calling class of the conversion service interface, obtain a bridging calling class, and make the preset conversion service interface call the bridging calling class to perform the conversion processing on the data in the data source table.

[0198] In an optional embodiment, the mapping module 51 is specifically configured to:

[0199] Based on the object-relational mapping rule, a class name corresponding to a table name of an input data source table is determined, and a class attribute name corresponding to a table field name of the data source is determined. Each attribute includes a table name and a table field name.

[0200] The table name and the corresponding class name, and the table field name and the corresponding class attribute name are mapped and associated by an annotation method.

[0201] In an optional embodiment, the rule agent module 52 is specifically configured to:

[0202] The object-relational mapping class is instantiated using a reflection mechanism of an object-oriented programming language, and the instantiated object-relational mapping class is subjected to attribute assignment.

[0203] The operation method of the data stream batch processing is woven into the object-relational mapping class after the attribute assignment by a dynamic proxy method, and a rule agent class with the operation method in the object-oriented programming language is obtained.

[0204] In an optional embodiment, the bridge construction module 53 is specifically configured to:

[0205] Based on the operation method of the rule agent class, the related construction method, assembly method and destruction method satisfying the preset conversion service conversion call are added by using a bridge mode;

[0206] Based on the rule agent class after the related construction method, assembly method and destruction method are added, a bridge call class is obtained.

[0207] In an optional embodiment, the device further includes an annotation module configured to:

[0208] If the input data source table does not satisfy the preset condition, annotation information is added to the data source table by an annotation method, and the annotation information is used to indicate that the SQL query statement is used for expression;

[0209] The data source table after the annotation information is added is input into a SQL parsing engine for conversion processing.

[0210] In the embodiments of the present application, the object-oriented idea is used through object relationship mapping, the data source table is mapped to an object form through metadata, and object operation of the object-oriented programming language is realized by converting the Flink SQL operation, that is, on the basis of the existing layered SQL level of the Flink stream batch data operation, an object relationship mapping layer operation is added through the object relationship mapping rule, so that when the business requirement of real-time computing stream or batch processing is implemented, the SQL script statement does not need to be defined, and the related entity class and attribute can be directly defined according to the business requirement, the structure of the data source table is described through the object relationship mapping class, and then the abstraction of the data source table is realized, so as to realize the conversion processing of the data in the data source table. Moreover, the data source table is mapped to an object form, the abstraction granularity is small, the later maintenance is facilitated, and the business model can be combined.

[0211] Based on the same inventive concept, the embodiments of the present application also provide an electronic device. Since the principle of the electronic device solving the problem is similar to the method, the implementation of the electronic device can be referred to the embodiments of the method, and the repeated parts will not be described herein.

[0212] Referring to Figure 6 As shown in the figure, the electronic device can include a processor 62 and a memory 61. The memory 61 provides the processor 62 with program instructions and data stored in the memory 61. In the embodiments of the present disclosure, the memory 61 can be used to store the program of the multimedia resource processing in the embodiments of the present disclosure.

[0213] The processor 62 processes the program instructions stored in the memory 61, and the processor 62 is used to execute the method in any method embodiment described above, for example Figure 1 The embodiments shown provide a data stream batch processing method.

[0214] The specific connection medium between the memory 61 and the processor 62 is not limited in the embodiments of the present disclosure. In the embodiments of the present disclosure Figure 6 , the memory 61 and the processor 62 are connected through a bus 63, and the bus 63 is represented by a thick line in Figure 6 , the connection mode between other components is only schematically described, and is not limited. The bus 63 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 6 , only one thick line is used, but it does not mean that there is only one bus or one type of bus.

[0215] The memory can include a Read-Only Memory (ROM) and a Random Access Memory (RAM), and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0216] The aforementioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc., and can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, etc.

[0217] The embodiments of the present disclosure further provide a computer storage medium, which stores a computer program. A processor of an electronic device reads the computer program from the computer readable storage medium, and the processor executes the computer program, so that the electronic device performs the energy-saving method of the data center in any of the above method embodiments.

[0218] In the specific implementation process, the computer storage medium can include a Universal Serial Bus Flash Drive (USB), a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various storage media that can store program codes.

[0219] Based on the same inventive concept as the above method embodiments, the embodiments of the present disclosure provide a computer program product, which includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the steps of any of the above data stream batch processing methods.

[0220] The computer program product can employ any combination of one or more computer-readable media or mediums. The computer-readable media or mediums can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0221] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable code.

[0222] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 The means for performing the function specified by one or more of the flow(s) or block(s) in the flowchart illustrations and / or block diagrams. Figure 1 The means for performing the function specified by one or more of the flow(s) or block(s) in the flowchart illustrations and / or block diagrams.

[0223] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams. Figure 1 The means for performing the function specified by one or more of the flow(s) or block(s) in the flowchart illustrations and / or block diagrams. Figure 1 The means for performing the function specified by one or more of the flow(s) or block(s) in the flowchart illustrations and / or block diagrams.

[0224] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams.Figure 1 one or more processes and / or functions specified in one or more blocks Figure 1 one or more processes and / or functions specified in one or more blocks

[0225] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A data stream batch processing method, characterized in that, include: Based on object-relational mapping rules, the input data source table is mapped to the corresponding object-relational mapping class, which refers to a class defined by an object-oriented programming language; Using the reflection mechanism of an object-oriented programming language, the object-relational mapping class is instantiated, and the instantiated object-relational mapping class is then assigned attribute values. By using dynamic proxies, data stream batch processing operation methods are woven into the object-relational mapping class after attribute assignment, thus obtaining a rule proxy class with operation methods in an object-oriented programming language. By adopting the bridge pattern, relevant constructors, assembly methods, and destructors that satisfy the preset conversion service conversion calls are added to the operation methods of the rule proxy class; Based on the rule proxy class with added constructors, assembly methods, and destructors, a bridge call class is obtained, which enables the preset conversion service interface to call the bridge call class to convert the data in the data source table.

2. The method according to claim 1, characterized in that, The method of mapping the input data source table to the corresponding object-relational mapping class based on object-relational mapping rules includes: Based on object-relational mapping rules, determine the class name corresponding to the table name of the input data source table, and the class attribute name corresponding to the table field name of the data source. The table name is mapped to the corresponding class name, and the table field name is mapped to the corresponding class attribute name using annotations.

3. The method according to claim 1, characterized in that, The method further includes: If the input data source table does not meet the preset conditions, annotation information is added to the data source table through annotation. The annotation information is used to indicate that an SQL query statement is used for expression. The data source table with the added annotation information is then input into the SQL parsing engine for conversion processing.

4. A data stream batch processing device, characterized in that, include: The mapping module is used to map the input data source table to the corresponding object-relational mapping class based on object-relational mapping rules. The object-relational mapping class refers to a class defined by an object-oriented programming language. The rule proxy module is used to instantiate the object-relational mapping class using the reflection mechanism of an object-oriented programming language, and to assign attribute values ​​to the instantiated object-relational mapping class. Through dynamic proxy, data flow batch processing operation methods are woven into the object-relational mapping class after attribute assignment to obtain a rule proxy class with operation methods in an object-oriented programming language. The bridging constructor module is used to add relevant constructors, assembly methods, and destructors to the operation methods of the rule proxy class, based on the bridging pattern, to satisfy the preset conversion service conversion call. Based on the rule proxy class with the added constructors, assembly methods, and destructors, a bridging call class is obtained, so that the preset conversion service interface calls the bridging call class to perform conversion processing on the data in the data source table.

5. The apparatus according to claim 4, characterized in that, The mapping module is specifically used for: Based on object-relational mapping rules, determine the class name corresponding to the table name of the input data source table, and the class attribute name corresponding to the table field name of the data source. The table name is mapped to the corresponding class name, and the table field name is mapped to the corresponding class attribute name using annotations.

6. The apparatus according to claim 4, characterized in that, The rule proxy module is specifically used for: Using the reflection mechanism of an object-oriented programming language, the object-relational mapping class is instantiated, and the instantiated object-relational mapping class is then assigned attribute values. By using dynamic proxies, data stream batch processing operation methods are woven into the object-relational mapping class after attribute assignment, thus obtaining a rule proxy class with operation methods in an object-oriented programming language.

7. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the method according to any one of claims 1 to 3.

Images