Method, device and electronic equipment for streaming data warehousing

Abstract

The application provides a stream data warehousing method and device and electronic equipment. The method comprises the following steps: obtaining a global variable configuration of a Flink task; when a stream data warehousing component is initialized, the global variable configuration is parsed to obtain a task configuration file; a SQL statement is transmitted to a Mapper class as a parameter to configure the stream data warehousing component; and the stream data warehousing is performed by using the configured stream data warehousing component. The method configures the global variable configuration of the Flink task, and uses the parameter transmission mode to configure the SQL statement, avoids the hard coding, and solves the problem of hard coding in the stream data warehousing by using the JDBC connector component provided by the Flink in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of databases, and in particular to a stream data warehousing method and device and electronic equipment. BACKGROUND

[0002] Most software applications or WEB systems currently implement front-end data display or back-end business functions by adding, deleting, querying or modifying data in the database, so after a general stream computing engine Flink is used to process raw real-time data streams, the result data after processing needs to be written into the database in a flexible and reliable manner.

[0003] In the above scenario, the Flink native JDBC connector component performs data warehousing by directly executing the DML statements (data manipulation language) defined in the code for the database. When the native JDBC connector component performs warehousing operations, if the DDL statement (data definition language) of the external database table has a primary key, the JDBC connector uses an update insertion method for warehousing, otherwise it uses a direct insertion method for warehousing.

[0004] Although the existing Flink native JDBC connector component implements the warehousing function of stream data, this component has the following disadvantages in complex business scenarios and actual development processes:

[0005] 1) There is a hard coding situation

[0006] In the use process, the database connection information and the SQL statement to be executed need to be defined in the code, and the mapping relationship between the fields in the data flowing from the previous component and the placeholders in the SQL statement needs to be set;

[0007] 2) Unable to manage database connections

[0008] In the use process, only the connection established when the JDBC connector component is created can be reused, and the database connections cannot be managed through a database connection pool to cope with complex scenarios such as database connection timeout, abnormal reconnection, connection recycling, etc. when connecting to the database;

[0009] 3) Single function and low expandability

[0010] Only simple insertion or update insertion operations can be performed. In actual business scenarios, the data warehousing strategy needs to consider factors such as data validity and timeliness to determine the warehousing strategy. There is no processing mechanism for warehousing exceptions. SUMMARY

[0011] The main purpose of the present application is to provide a stream data warehousing method, device and electronic equipment, so as to at least solve the problem of hard coding in the stream data warehousing of the JDBC connector component provided by Flink natively.

[0012] In order to achieve the above purpose, according to one aspect of the present application, a stream data warehousing method is provided, comprising: obtaining a global variable configuration of a Flink task; when a stream data warehousing component is initialized, parsing the global variable configuration to obtain a task configuration file, the task configuration file comprising at least one of: database connection information, database connection pool parameters, Mybatis configuration information, the location of a Mapper class package, and a SQL statement; passing the SQL statement as a parameter to the Mapper class to configure the stream data warehousing component; and using the configured stream data warehousing component to perform stream data warehousing.

[0013] Optionally, after the global variable configuration is parsed to obtain the task configuration file when the data warehousing component is initialized, the method further comprises: executing an open method in a SinkFunction interface to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package.

[0014] Optionally, after the open method in the SinkFunction interface is executed to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package, the method further comprises: constructing a database connection pool factory using an api provided by a Mybatis framework according to the read database connection pool parameters; constructing a Mybatis session factory using the api provided by the Mybatis framework according to the read Mybatis configuration information; and registering the Mapper class to the Mybatis session factory based on the location of the Mapper class package using the api provided by the Mybatis framework.

[0015] Optionally, the method further comprises: after the initialization of the stream data warehousing component is completed, executing an operation in an invoke method in the SinkFunction interface that needs to access a database through the Mybatis framework according to business needs.

[0016] Optionally, after the stream data warehousing component is initialized, the operation of accessing the database through the Mybatis framework in the invoke in the SinkFunction interface is executed according to the business requirement, including: before each execution of the operation of accessing the database, controlling the database transaction to be in an open state; after each execution of the operation of accessing the database, in the case that no exception occurs in the execution process, committing the database transaction, and in the case that an exception occurs in the execution process, rolling back the database transaction.

[0017] Optionally, before each execution of the operation of accessing the database, controlling the database transaction to be in an open state, including: before each execution of the operation of accessing the database, controlling the database transaction to be in the open state through the way of JDK dynamic proxy; after each execution of the operation of accessing the database, in the case that no exception occurs in the execution process, committing the database transaction, and in the case that an exception occurs in the execution process, rolling back the database transaction, including: after each execution of the operation of accessing the database, in the case that no exception occurs in the execution process, committing the database transaction, and in the case that an exception occurs in the execution process, rolling back the database transaction through the way of the JDK dynamic proxy.

[0018] Optionally, the method further includes: executing the SQL statement to execute the operation of accessing the database.

[0019] Optionally, after the global variable configuration is parsed to obtain the task configuration file when the stream data warehousing component is initialized, the method further includes: passing the task configuration file to the ExecutionConfig interface.

[0020] According to another aspect of the present application, a stream data warehousing device is provided, including: an acquisition unit configured to acquire a global variable configuration of a Flink task; a parsing unit configured to parse the global variable configuration to obtain a task configuration file when a stream data warehousing component is initialized, the task configuration file including at least one of: database connection information, database connection pool parameters, Mybatis configuration information, a location of a Mapper class, and a SQL statement; a passing unit configured to pass the SQL statement as a parameter to the Mapper class to configure the stream data warehousing component; and a warehousing unit configured to perform stream data warehousing by using the configured stream data warehousing component.

[0021] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any one of the stream data warehousing methods.

[0022] According to the technical solution of the present application, the stream data warehousing method comprises: obtaining a global variable configuration of a Flink task; parsing the global variable configuration to obtain a task configuration file when a stream data warehousing component is initialized, the task configuration file comprising at least one of: database connection information, database connection pool parameters, Mybatis configuration information, the location of a Mapper class package, and a SQL statement; passing the SQL statement as a parameter to the Mapper class to configure the stream data warehousing component; and using the configured stream data warehousing component to perform stream data warehousing. The method configures the global variable configuration of the Flink task, and uses a parameter passing manner to configure the SQL statement, thereby avoiding hard coding and solving the problem of hard coding in the stream data warehousing of the Flink native provided JDBC connector component. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which form a part of the specification, are included to provide a further understanding of the application and are incorporated herein in conjunction with the description of the application. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the application. In the drawings:

[0024] Figure 1 A hardware structure block diagram of a mobile terminal for performing a stream data warehousing method according to an embodiment of the present application is shown;

[0025] Figure 2 A flowchart of a stream data warehousing method according to an embodiment of the present application is shown;

[0026] Figure 3 A flowchart of another method of a stream data warehousing method according to an embodiment of the present application is shown;

[0027] Figure 4 A structure block diagram of a stream data warehousing apparatus according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0029] In order to make the person skilled in the art better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] For the convenience of description, the following describes some nouns or terms related to the embodiments of the present application:

[0032] Flink: Apache Flink is an open source stream processing framework developed by Apache Software Foundation, and its core is a distributed stream data stream engine written in Java and Scala. Flink executes any stream data program in a data parallel and pipelined manner, and the pipelined runtime system of Flink can execute batch processing and stream processing programs. In addition, the runtime of Flink itself also supports the execution of iterative algorithms.

[0033] Mybatis: MyBatis is a general-purpose persistence layer framework that supports customized SQL, stored procedures and advanced mapping. It avoids almost all JDBC code and manual setting of parameters and retrieval of result sets, and can use simple XML or annotations to configure and map native information, mapping interfaces and Java POJOs (Plain Ordinary Java Object) into records in the database.

[0034] JDK dynamic proxy: proxy mode refers to providing a proxy for the agent object to control its access to the target object. In some cases, an object is not suitable or cannot directly reference another object, and the proxy object can act as an intermediary between the agent object and the target object. Using a proxy object, the main business logic of the agent object can be enhanced without modifying the agent object. JDK dynamic proxy is implemented based on the reflection mechanism of Java. Dynamic creation of proxy objects using interfaces and classes in JDK requires that the target object implement an interface. Since JDK1.3, the java language provides three classes in the java.lang.reflect package to support the proxy mode Proxy, Method and InovcationHandler.

[0035] As introduced in the background, the existing JDBC connector component provided by Flink natively has the problems of hard coding, inability to manage database connections, and single function and poor expansibility. To solve the problem of hard coding of the existing JDBC connector component provided by Flink natively for streaming data warehousing, the embodiments of the present application provide a streaming data warehousing method, device and electronic equipment.

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application.

[0037] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking the case of running on a mobile terminal, Figure 1 is a hardware structure block diagram of a mobile terminal of a streaming data warehousing method according to an embodiment of the present application. As shown in Figure 1 , the mobile terminal can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned mobile terminal can also include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 the structure shown is only schematic, which does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal can also include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0038] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] This embodiment provides a method for streaming data import that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] Figure 2 This is a flowchart of a streaming data import method according to an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps:

[0041] Step S201: Obtain the global variable configuration of the Flink task;

[0042] Step S202: During the initialization of the streaming data ingestion component, the above global variable configuration is parsed to obtain the task configuration file. The task configuration file includes at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, the location of the Mapper class in the package, and SQL statements.

[0043] The method further includes: passing the task configuration file to an ExecutionConfig interface after parsing the global variable configuration when the stream data warehousing component is initialized to obtain the task configuration file. The ExecutionConfig interface can be accessed in all components, so the task configuration file passed after task submission can be accessed by all component initialization. The method is implemented based on a SinkFunction interface provided by Flink natively, can be seamlessly integrated with other components or processing mechanisms in the Flink framework, and reduces the implementation complexity.

[0044] Step S203: passing the SQL statement as a parameter to the Mapper class to configure the stream data warehousing component.

[0045] Specifically, Mybatis supports two ways of configuring SQL, i.e., an XML way and an annotation way in a Mapper class (defining an interface for database operation). The present solution adopts the annotation way to configure, and passes the SQL obtained by reading the configuration as a parameter to the method defined in the Mapper class.

[0046] The method further includes: executing an open method in the SinkFunction interface to read the database connection pool parameters, the Mybatis configuration information and the position of the Mapper class after parsing the global variable configuration when the data warehousing component is initialized to obtain the task configuration file. Specifically, the method further includes parsing Druid connection pool configuration. The method is implemented by using the SinkFunction interface of Flink natively, and can be seamlessly integrated with other components in the Flink task.

[0047] The method further includes: Figure 3

[0048] Step S301: constructing a database connection pool factory by using an api provided by the Mybatis framework according to the read database connection pool parameters;

[0049] Step S302: constructing a Mybatis session factory by using the api provided by the Mybatis framework according to the read Mybatis configuration information;

[0050] ​Step S303, the Mapper class is registered to the Mybatis session factory based on the location of the package where the Mapper class is located by using the api provided by the Mybatis framework.

[0051] After the above steps are performed, the integration of the database connection pool and the instantiation of the Mybatis session factory are finally completed.

[0052] In an example, the method further includes: after the initialization of the stream data warehousing component is completed, performing an operation in the invoke of the SinkFunction interface that needs to access the database through the Mybatis framework according to business needs.

[0053] In each execution of the operation of accessing the database, the database transaction is controlled to be in an open state; specifically, through the way of JDK dynamic proxy, the database transaction is controlled to be in the open state before each execution of the operation of accessing the database.

[0054] In each execution of the operation of accessing the database, the database transaction is controlled to be in an open state; specifically, through the way of JDK dynamic proxy, the database transaction is controlled to be in the open state before each execution of the operation of accessing the database.

[0055] The above scheme controls the database transaction to be in an open state before and after the execution of the operation of accessing the database through the way of dynamic proxy. The database connection pool and the transaction automatic opening and submission mechanism are integrated, and the underlying technical details such as database connection management and transaction management are shielded, which is conducive to focusing on business implementation in the development process.

[0056] In an example, the specific process of the stream data warehousing method is as follows Figure 3As shown, firstly, the global variable configuration of the Flink task is acquired, then the Mybatis configuration information and the Druid connection pool configuration information are parsed, then the database connection pool factory and the Mybatis session factory are constructed, then the position of the package where the Mapper class defining the database operation is located is parsed, then the Mapper class is registered in the Mybatis session factory, the transaction is opened through the Mybatis session factory, the Mapper class instance is acquired through the Mybatis session factory, the Mapper class method is executed through the Mapper class instance, then it is judged whether an exception occurs in the execution process, in the case where an exception occurs in the execution process, the transaction is rolled back through the Mybatis session factory, and the exception is handled, and in the case where no exception occurs in the execution process, the transaction is committed through the Mybatis session factory.

[0057] Specifically, the invoke method can not only execute the database access operation through the Mybatis session factory, but also execute other non-database access operations according to business needs.

[0058] In one scheme, the above method further includes: executing the above SQL statement to execute the operation of accessing the database. The SQL statement for operating the database is decoupled from the Flink task component, the development complexity is reduced, and the maintenance of the SQL in the later period is facilitated.

[0059] In step S204, the configured stream data warehousing component is used to perform stream data warehousing.

[0060] When the task is stopped, the Flink warehousing component is closed.

[0061] The above stream data warehousing method of the present application includes: acquiring the global variable configuration of the Flink task; when the stream data warehousing component is initialized, the global variable configuration is parsed to obtain a task configuration file, the task configuration file includes at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, the position of the package where the Mapper class is located, and a SQL statement; the SQL statement is passed into the Mapper class as a parameter to configure the stream data warehousing component; and the configured stream data warehousing component is used to perform stream data warehousing. This method configures the global variable configuration of the Flink task, and uses the parameter passing mode to realize the configuration of the SQL statement, avoids hard coding, and solves the problem of hard coding in the stream data warehousing of the Flink native provided JDBC connector component in the prior art.

[0062] It is noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0063] The embodiment of the present application further provides a stream data warehousing device. It should be noted that the stream data warehousing device of the embodiment of the present application can be used to execute the stream data warehousing method provided by the embodiment of the present application. The device is used to realize the above-mentioned embodiment and preferred embodiment, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the device described in the following embodiment is preferably realized in software, realization of hardware, or a combination of software and hardware, is also possible and is contemplated.

[0064] The stream data warehousing device provided by the embodiment of the present application is described below.

[0065] Figure 4 is a schematic diagram of the stream data warehousing device according to the embodiment of the present application. As shown in Figure 4 the device includes: an acquisition unit 10 configured to acquire a global variable configuration of a Flink task; an analysis unit 20 configured to analyze the global variable configuration when a stream data warehousing component is initialized, to obtain a task configuration file, the task configuration file including at least one of: database connection information, database connection pool parameters, Mybatis configuration information, a location of a Mapper class, and a SQL statement; a delivery unit 30 configured to deliver the SQL statement as a parameter to the Mapper class, to configure the stream data warehousing component; and a warehousing unit 40 configured to use the configured stream data warehousing component to perform stream data warehousing.

[0066] As an optional solution, the device further includes a reading unit configured to execute an open device in a SinkFunction interface, and read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class. The method is completed using the SinkFunction interface of the Flink native, and can be seamlessly integrated with other components in the Flink task.

[0067] In an alternative solution, the device further comprises a connection module, a construction module and a registration module, the connection module is configured to construct a database connection pool factory by using an API provided by the Mybatis framework according to the read database connection pool parameters; the construction module is configured to construct a Mybatis session factory by using an API provided by the Mybatis framework according to the read Mybatis configuration information; and the registration module is configured to register the Mapper class to the Mybatis session factory based on the location of the Mapper class by using an API provided by the Mybatis framework. The seamless integration between different components in the Flink task is achieved.

[0068] In the embodiment, the device further comprises an access unit, which is configured to perform an operation of accessing a database by using the Mybatis framework in the invoke in the SinkFunction interface according to business needs after the initialization of the stream data warehousing component is completed. The integration process of the Mybtatis framework and the database connection pool is achieved.

[0069] For example, the access unit comprises a control unit and a first execution unit, the control unit is configured to control the database transaction to be in an open state before each execution of the operation of accessing the database, and the first execution unit is configured to commit the database transaction in the case where no exception occurs in the execution process after each execution of the operation of accessing the database, and roll back the database transaction in the case where an exception occurs in the execution process. The database transaction is automatically opened and committed or rolled back before and after the execution of the Mapper class method operation database in the manner of dynamic proxy, and the database connection pool and the transaction automatic opening and commitment mechanism are integrated.

[0070] In an alternative embodiment, the control unit comprises a control module, the control module is configured to control the database transaction to be in the open state by using the JDK dynamic proxy before each execution of the operation of accessing the database, and the first execution unit comprises an execution module, the execution module is configured to commit the database transaction in the case where no exception occurs in the execution process after each execution of the operation of accessing the database by using the JDK dynamic proxy, and roll back the database transaction in the case where an exception occurs in the execution process. The database connection pool and the transaction automatic opening and commitment mechanism are integrated, and the underlying technical details such as database connection management and transaction management are shielded, which is conducive to focusing on business implementation in the development process.

[0071] In the embodiment, the device further comprises a second execution unit, configured to execute the SQL statement to perform the operation of accessing the database. The SQL statement for operating the database is decoupled from the Flink task component, the development complexity is reduced, and the SQL maintenance is facilitated.

[0072] In an example, the device further comprises a delivery module, configured to deliver the task configuration file to an ExecutionConfig interface. After the task is submitted, all components can be initialized to access.

[0073] The device for storing stream data in the database according to the application comprises: an acquisition unit 10, configured to acquire a global variable configuration of a Flink task; an analysis unit 20, configured to analyze the global variable configuration to obtain a task configuration file when a stream data storage component is initialized, wherein the task configuration file comprises at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, a location of a Mapper class, and a SQL statement; a delivery unit 30, configured to deliver the SQL statement as a parameter to the Mapper class to configure the stream data storage component; and a storage unit 40, configured to store stream data by using the configured stream data storage component. The device acquires the global variable configuration of the Flink task, and configures the SQL statement by using the parameter delivery mode, thereby avoiding hard coding and solving the problem of hard coding in the stream data storage by using the JDBC connector component provided by the Flink native technology.

[0074] The device for storing stream data in the database comprises a processor and a memory, the acquisition unit and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory. The modules are located in the same processor, or the modules are located in different processors in any combination.

[0075] The processor comprises a core, and the core retrieves the corresponding program unit from the memory. The core can be one or more, and the problem of hard coding in the stream data storage by using the JDBC connector component provided by the Flink native technology is solved by adjusting the core parameters.

[0076] The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0077] The embodiment of the application provides a computer readable storage medium, the computer readable storage medium comprises a stored program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the stream data warehousing method when the program runs.

[0078] Specifically, the stream data warehousing method comprises the following steps.

[0079] In step S201, a global variable configuration of a Flink task is acquired.

[0080] Specifically, the stream data warehousing method is realized based on a SinkFunction interface provided by Flink natively, can be seamlessly fused with other components or processing mechanisms in the Flink framework, and simultaneously reduces the complexity of implementation.

[0081] In step S202, when a stream data warehousing component is initialized, the global variable configuration is parsed to obtain a task configuration file, and the task configuration file comprises at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, the location of a Mapper class, and a SQL statement.

[0082] Specifically, Mybatis supports two ways of configuring SQL, that is, an XML mode and an annotation mode in a Mapper class (defines an interface for database operation), and the scheme adopts the annotation mode to configure, and after SQL is obtained by reading the configuration, the SQL is passed into a method defined in the Mapper class as a parameter.

[0083] In step S203, the SQL statement is passed into the Mapper class as a parameter to configure the stream data warehousing component.

[0084] Specifically, the method of the Mapper class is executed in a way that a database transaction is automatically opened and committed or rolled back before and after the operation of the database, the database connection pool and the transaction automatic opening and committing mechanism are integrated, the underlying technical details such as database connection management and transaction management are shielded, and the business implementation can be concentrated in the development process.

[0085] In step S204, the stream data warehousing is performed by using the configured stream data warehousing component.

[0086] Specifically, the step realizes the decoupling of the SQL statement for operating the database and the Flink task component, reduces the development complexity, and is beneficial to the maintenance of the SQL later.

[0087] Optionally, after the global variable configuration is parsed at the initialization of the data warehousing component and the task configuration file is obtained, the method further comprises: executing an open method in the SinkFunction interface to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package.

[0088] Optionally, after the open method in the SinkFunction interface is executed to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package, the method further comprises: constructing a database connection pool factory using an api provided by the Mybatis framework according to the read database connection pool parameters; constructing a Mybatis session factory using the api provided by the Mybatis framework according to the read Mybatis configuration information; and registering the Mapper class to the Mybatis session factory based on the location of the Mapper class package using the api provided by the Mybatis framework.

[0089] Optionally, the method further comprises: after the initialization of the stream data warehousing component is completed, executing an operation in the invoke in the SinkFunction interface that needs to access the database through the Mybatis framework according to business needs.

[0090] Optionally, after the initialization of the stream data warehousing component is completed, executing an operation in the invoke in the SinkFunction interface that needs to access the database through the Mybatis framework according to business needs comprises: before each execution of the operation of accessing the database, controlling a database transaction to be in an open state; and after each execution of the operation of accessing the database, committing the database transaction in the case that an execution process does not occur abnormally, and rolling back the database transaction in the case that the execution process occurs abnormally.

[0091] Optionally, before each execution of the operation of accessing the database, controlling the database transaction to be in the open state comprises: before each execution of the operation of accessing the database, controlling the database transaction to be in the open state through a JDK dynamic proxy; and after each execution of the operation of accessing the database, committing the database transaction in the case that an execution process does not occur abnormally, and rolling back the database transaction in the case that the execution process occurs abnormally, comprises: after each execution of the operation of accessing the database, committing the database transaction in the case that the execution process does not occur abnormally, and rolling back the database transaction in the case that the execution process occurs abnormally through the JDK dynamic proxy.

[0092] Optionally, the method further comprises: executing the SQL statement to perform the operation of accessing the database.

[0093] Optionally, after the global variable configuration is parsed when the stream data warehousing component is initialized to obtain the task configuration file, the method further comprises: passing the task configuration file to the ExecutionConfig interface.

[0094] Embodiments of the application provide a processor for running a program, wherein the processor is used for running the stream data warehousing method.

[0095] Specifically, the stream data warehousing method comprises:

[0096] In step S201, a global variable configuration of a Flink task is acquired.

[0097] Specifically, the SinkFunction interface provided by Flink is used for implementation, and the method can be seamlessly integrated with other components or processing mechanisms in the Flink framework, and the complexity of implementation is reduced.

[0098] In step S202, the global variable configuration is parsed when the stream data warehousing component is initialized to obtain a task configuration file, and the task configuration file comprises at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, the location of a Mapper class, and a SQL statement.

[0099] Specifically, Mybatis supports two SQL configuration modes, i.e., an XML mode and an annotation mode in a Mapper class (an interface for defining database operations), and the method adopts the annotation mode to configure, and the SQL is obtained by reading the configuration and is passed into a method defined in the Mapper class as a parameter.

[0100] In step S203, the SQL statement is passed into the Mapper class as a parameter to configure the stream data warehousing component.

[0101] Specifically, the method is executed by using a dynamic proxy to automatically open, commit or roll back a database transaction before and after a Mapper class method is used to operate a database. The method integrates a database connection pool and a transaction automatic opening and committing mechanism, and shields underlying technical details such as database connection management and transaction management, which is beneficial to focusing on business implementation in the development process.

[0102] In step S204, the stream data warehousing component is configured, and stream data warehousing is performed.

[0103] Specifically, the above steps achieve decoupling of SQL statements for operating a database and Flink task components, reduce development complexity, and facilitate later maintenance of SQL.

[0104] Optionally, after the global variable configuration is parsed and the task configuration file is obtained when the data warehousing component is initialized, the method further includes: executing an open method in the SinkFunction interface to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package.

[0105] Optionally, after the open method in the SinkFunction interface is executed to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package, the method further includes: constructing a database connection pool factory by using an api provided by the Mybatis framework according to the read database connection pool parameters; constructing a Mybatis session factory by using the api provided by the Mybatis framework according to the read Mybatis configuration information; and registering the Mapper class to the Mybatis session factory based on the location of the Mapper class package by using the api provided by the Mybatis framework.

[0106] Optionally, the method further includes: after the initialization of the stream data warehousing component is completed, executing an operation in the invoke in the SinkFunction interface that needs to access a database through the Mybatis framework according to business needs.

[0107] Optionally, after the initialization of the stream data warehousing component is completed, the operation in the invoke in the SinkFunction interface that needs to access a database through the Mybatis framework according to business needs includes: before each execution of the operation of accessing the database, controlling a database transaction to be in an open state; and after each execution of the operation of accessing the database, committing the database transaction in a case where an execution process does not occur abnormally, and rolling back the database transaction in a case where the execution process occurs abnormally.

[0108] Optionally, before each execution of the operation of accessing the database, the database transaction is controlled to be in an open state, including: before each execution of the operation of accessing the database, the database transaction is controlled to be in the open state by means of a JDK dynamic agent; after each execution of the operation of accessing the database, if an execution process does not occur abnormally, the database transaction is committed, and if the execution process occurs abnormally, the database transaction is rolled back, including: after each execution of the operation of accessing the database, if the execution process does not occur abnormally, the database transaction is committed, and if the execution process occurs abnormally, the database transaction is rolled back by means of the JDK dynamic agent.

[0109] Optionally, the method further includes: executing the SQL statement to perform the operation of accessing the database.

[0110] Optionally, after the global variable configuration is parsed to obtain the task configuration file when the stream data warehousing component is initialized, the method further includes: passing the task configuration file to an ExecutionConfig interface.

[0111] An embodiment of the present application provides a device, the device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements at least the following steps when executing the program:

[0112] Step S201: global variable configuration of a Flink task is acquired;

[0113] Step S202: the global variable configuration is parsed to obtain a task configuration file when a stream data warehousing component is initialized, and the task configuration file includes at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, a location of a Mapper class, and a SQL statement;

[0114] Step S203: the SQL statement is passed to the Mapper class as a parameter to configure the stream data warehousing component;

[0115] Step S204: the stream data warehousing component after the configuration is used to perform stream data warehousing.

[0116] The device herein can be a server, a PC, a PAD, a mobile phone, or the like.

[0117] Optionally, after the global variable configuration is parsed to obtain the task configuration file when the data warehousing component is initialized, the method further includes: executing an open method in a SinkFunction interface to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class.

[0118] Optionally, after executing the open method in the SinkFunction interface, reading the database connection pool parameters, the Mybatis configuration information, and the location of the package of the Mapper class, the method further comprises: constructing a database connection pool factory using the api provided by the Mybatis framework according to the read database connection pool parameters; constructing a Mybatis session factory using the api provided by the Mybatis framework according to the read Mybatis configuration information; and registering the Mapper class to the Mybatis session factory based on the location of the package of the Mapper class using the api provided by the Mybatis framework.

[0119] Optionally, the method further comprises: after the initialization of the stream data warehousing component is completed, performing the operation of accessing the database through the Mybatis framework in the invoke in the SinkFunction interface according to business needs.

[0120] Optionally, after the initialization of the stream data warehousing component is completed, performing the operation of accessing the database through the Mybatis framework in the invoke in the SinkFunction interface according to business needs comprises: before each execution of the operation of accessing the database, controlling the database transaction to be in an open state; and after each execution of the operation of accessing the database, committing the database transaction in the case that no exception occurs in the execution process, and rolling back the database transaction in the case that an exception occurs in the execution process.

[0121] Optionally, before each execution of the operation of accessing the database, controlling the database transaction to be in an open state comprises: before each execution of the operation of accessing the database, controlling the database transaction to be in the open state through the way of JDK dynamic proxy; and after each execution of the operation of accessing the database, committing the database transaction in the case that no exception occurs in the execution process, and rolling back the database transaction in the case that an exception occurs in the execution process, comprises: after each execution of the operation of accessing the database, committing the database transaction in the case that no exception occurs in the execution process, and rolling back the database transaction in the case that an exception occurs in the execution process through the way of the JDK dynamic proxy.

[0122] Optionally, the method further comprises: executing the SQL statement to perform the operation of accessing the database.

[0123] Optionally, after the global variable configuration is parsed at the initialization of the streaming data warehousing component to obtain the task configuration file, the method further includes: passing the task configuration file to the ExecutionConfig interface.

[0124] The application also provides a computer program product adapted to execute a program that initializes at least the following method steps when executed on a data processing device: step S201, obtaining a global variable configuration of a Flink task;

[0125] Step S202, parsing the global variable configuration at the initialization of the streaming data warehousing component to obtain a task configuration file, the task configuration file including at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, the location of a Mapper class package, and a SQL statement.

[0126] Step S203, passing the SQL statement as a parameter to the Mapper class to configure the streaming data warehousing component.

[0127] Step S204, using the configured streaming data warehousing component to perform streaming data warehousing.

[0128] Optionally, after the global variable configuration is parsed at the initialization of the data warehousing component to obtain the task configuration file, the method further includes: executing the open method in the SinkFunction interface to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package.

[0129] Optionally, after the open method in the SinkFunction interface is executed to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class package, the method further includes: constructing a database connection pool factory using the api provided by the Mybatis framework according to the read database connection pool parameters; constructing a Mybatis session factory using the api provided by the Mybatis framework according to the read Mybatis configuration information; and registering the Mapper class to the Mybatis session factory based on the location of the Mapper class package using the api provided by the Mybatis framework.

[0130] Optionally, the method further includes: after the initialization of the streaming data warehousing component is completed, executing the invoke method in the SinkFunction interface according to business needs, which requires accessing the database through the Mybatis framework.

[0131] Optionally, after the initialization of the stream data warehousing component is completed, the operation of accessing the database through the Mybatis framework in the invoke of the SinkFunction interface is executed according to the business needs, including: before each execution of the operation of accessing the database, controlling the database transaction to be in an open state; after each execution of the operation of accessing the database, in the case that no exception occurs in the execution process, committing the database transaction, and in the case that an exception occurs in the execution process, rolling back the database transaction.

[0132] Optionally, before each execution of the operation of accessing the database, controlling the database transaction to be in an open state, including: through the way of JDK dynamic proxy, before each execution of the operation of accessing the database, controlling the database transaction to be in the open state; after each execution of the operation of accessing the database, in the case that no exception occurs in the execution process, committing the database transaction, and in the case that an exception occurs in the execution process, rolling back the database transaction, including: through the way of the JDK dynamic proxy, after each execution of the operation of accessing the database, in the case that no exception occurs in the execution process, committing the database transaction, and in the case that an exception occurs in the execution process, rolling back the database transaction.

[0133] Optionally, the method further includes: executing the SQL statement to execute the operation of accessing the database.

[0134] Optionally, after the global variable configuration is parsed when the stream data warehousing component is initialized, and the task configuration file is obtained, the method further includes: passing the task configuration file to the ExecutionConfig interface.

[0135] Obviously, those skilled in the art should understand that each module or each step of the present application described above can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, which can be realized by program code executable by a computing device, so that they can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0136] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0137] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0138] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0139] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0140] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0141] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile or non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0142] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0143] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0144] From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

[0145] 1) The above-mentioned stream data warehousing method of the present application comprises: obtaining the global variable configuration of the Flink task; parsing the global variable configuration when the stream data warehousing component is initialized to obtain the task configuration file, the task configuration file comprising at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, the location of the Mapper class package, SQL statement; passing the SQL statement as a parameter to the Mapper class to configure the stream data warehousing component; using the configured stream data warehousing component to perform stream data warehousing. This method configures the global variable configuration of the Flink task, and uses the parameter passing method to configure the SQL statement, avoids hard coding, and solves the problem of hard coding in the stream data warehousing of the Flink native provided JDBC connector component in the prior art.

[0146] 2) The above-mentioned streaming data warehousing device of the present application, the device comprises: an acquisition unit 10 for acquiring the global variable configuration of the Flink task; an analysis unit 20 for analyzing the global variable configuration to obtain a task configuration file when the streaming data warehousing component is initialized, the task configuration file comprising at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, the location of the Mapper class package, and SQL statements; a transmission unit 30 for transmitting the SQL statements as parameters to the Mapper class to configure the streaming data warehousing component; and a warehousing unit 40 for using the configured streaming data warehousing component to perform streaming data warehousing. The device configures the global variable configuration of the Flink task, and uses the parameter transmission mode to realize the configuration of the SQL statements, avoids hard coding, and solves the problem of hard coding in the streaming data warehousing of the Flink native provided JDBC connector component in the prior art.

[0147] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for streaming data import, characterized in that, The method comprises: obtaining a global variable configuration of a Flink task; when a streaming data warehousing component is initialized, parsing the global variable configuration to obtain a task configuration file, wherein the task configuration file comprises at least one of the following: database connection information, database connection pool parameters, MyBatis configuration information, a location of a Mapper class, and a SQL statement; passing the SQL statement as a parameter to the Mapper class to configure the streaming data warehousing component; using the configured streaming data warehousing component to perform streaming data warehousing; after the global variable configuration is parsed to obtain the task configuration file when the data warehousing component is initialized, the method further comprises: executing an open method in a SinkFunction interface to read the database connection pool parameters, the MyBatis configuration information, and the location of the Mapper class; after the open method in the SinkFunction interface is executed to read the database connection pool parameters, the MyBatis configuration information, and the location of the Mapper class, the method further comprises: constructing a database connection pool factory using an API provided by a MyBatis framework according to the read database connection pool parameters; constructing a MyBatis session factory using an API provided by the MyBatis framework according to the read MyBatis configuration information; registering the Mapper class to the MyBatis session factory based on the location of the Mapper class using an API provided by the MyBatis framework.

2. The method of claim 1, wherein, The method further comprises: after the streaming data warehousing component is initialized, performing an operation that needs to access a database through a MyBatis framework in an invoke method in the SinkFunction interface according to business needs.

3. The method of claim 2, wherein, after the streaming data warehousing component is initialized, performing an operation that needs to access a database through a MyBatis framework in an invoke method in the SinkFunction interface according to business needs, comprising: controlling a database transaction to be in an open state before each execution of the operation of accessing the database; after each execution of the operation of accessing the database, committing the database transaction in a case where no exception occurs in an execution process, and rolling back the database transaction in a case where an exception occurs in the execution process.

4. The method of claim 3, wherein controlling a database transaction to be in an open state before each execution of the operation of accessing the database, comprising: controlling the database transaction to be in the open state before each execution of the operation of accessing the database through a JDK dynamic proxy; after each execution of the operation of accessing the database, committing the database transaction in a case where no exception occurs in an execution process, and rolling back the database transaction in a case where an exception occurs in the execution process, comprising: Commit the database transaction after each execution of the operation of accessing the database in a case where no exception occurs in the execution process, and roll back the database transaction in a case where an exception occurs in the execution process by means of the JDK dynamic proxy.

5. The method of claim 4, wherein, The method further includes: Executing the SQL statement to perform the operation of accessing the database.

6. The method according to any one of claims 1 to 3, characterized in that, After parsing the global variable configuration to obtain a task configuration file when a stream data warehousing component is initialized, the method further includes: Passing the task configuration file to an ExecutionConfig interface.

7. A stream data warehousing apparatus characterized by comprising: Comprise: An acquisition unit, configured to acquire a global variable configuration of a Flink task; A parsing unit, configured to parse the global variable configuration to obtain a task configuration file when a stream data warehousing component is initialized, the task configuration file comprising at least one of the following: database connection information, database connection pool parameters, Mybatis configuration information, a location of a Mapper class, and a SQL statement; A passing unit, configured to pass the SQL statement as a parameter to the Mapper class to configure the stream data warehousing component; A warehousing unit, configured to perform stream data warehousing by using the configured stream data warehousing component. The apparatus further comprises: A reading unit, configured to read the database connection pool parameters, the Mybatis configuration information, and the location of the Mapper class by executing an open method in a SinkFunction interface; The apparatus further comprises: A connection module, configured to construct a database connection pool factory by using an api provided by a Mybatis framework according to the read database connection pool parameters; A construction module, configured to construct a Mybatis session factory by using the api provided by the Mybatis framework according to the read Mybatis configuration information; A registration module, configured to register the Mapper class to the Mybatis session factory based on the location of the Mapper class by using the api provided by the Mybatis framework.

8. An electronic device, comprising: Comprise: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise a stream data warehousing method according to any one of claims 1 to 6.

Images